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ABSTRACT 


This  study  analyzes  the  impact  various  capabilities  have  on  intelligence  gathering 
missions  for  a  Marine  Expeditionary  Brigade  (MEB)  commander’s  2015  unmanned  aerial 
vehicle  (UAV).  The  Marine  Corps  Warlighting  Lab  (MCWL)  is  developing 
requirements  for  an  intelligence,  surveillance,  and  reconnaissance  (ISR)  UAV  that 
supports  rapid  planning  and  decision  making  for  multiple  concurrent  operations  and 
facilitates  maneuver  and  precision  engagement.  Additionally,  acquisition  of  a  2008 
Pioneer  replacement  is  underway  at  Marine  Corps  Systems  Command 
(MARCORSYSCOM).  The  importance  of  various  capabilities  for  this  replacement  UAV 
presently  lacks  quantitative  analysis.  Through  modeling,  agent-based  simulation,  and 
data  mining,  this  study  explores  the  validity  of  current  requirements  and  provides  insights 
into  the  importance  of  various  UAV  characteristics,  such  as  airspeed,  endurance,  sweep 
width,  and  sensor  capability.  The  results  have  design  consequences  for  MCWL’s  Fleet 
Battle  Experiment  Sea  Viking  20XX,  its  largest  annual  experiment,  and  provide  key 
parameters  for  physics-based  simulations  such  as  COMBAT  XXI.  The  advantage  of 
tactical  routing,  a  seven  hour  (or  greater)  on  station  time,  a  minimum  4,500  meter  sweep 
width,  and  a  probability  of  classification  of  at  least  0.4  are  verified  for  the  Sea  Viking 
scenario.  This  analysis  indicates  that  a  UAV  in  this  scenario  does  not  need  to  travel  in 
excess  of  200  knots. 
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EXECUTIVE  SUMMARY 


Unmanned  Aerial  Vehicles  (UAVs)  are  a  key  component  of  today’s  Intelligence, 
Surveillance  and  Reconnaissance  (ISR)  missions.  UAVs  provide  intelligence,  a  dynamic 
retasking  capability,  and  real-time  video  imagery.  The  United  States  Marine  Corps  is 
currently  reviewing  its  UAV  employment.  During  Operation  Iraqi  Freedom  (OIF),  the 
Marine  Expeditionary  Brigade  (MEB)  commander  utilized  two  squadrons  of  Pioneer 
UAVs.  Each  squadron  provided  the  capability  to  fly  one  UAV  for  up  to  six  hours  with  a 
maximum  range  of  over  170  nautical  miles.  The  Pioneer  can  be  relieved  on  station  twice 
with  current  assets  and  manpower,  achieving  a  total  coverage  of  18-hours  during  surge 
operations  with  the  current  force  structure  in  each  squadron.  The  Marine  Corps  desires  to 
improve  this  capability  in  an  efficient  and  effective  manner. 

The  planned  Pioneer  replacement,  the  Vertical  Unmanned  Aerial  Vehicle,  is  in 
the  beginning  stages  of  conceptualization  and  currently  has  an  Initial  Capabilities 
Document.  It  is  expected  to  proceed  through  the  design  and  acquisition  process  for  fleet 
use  in  2008.  Marine  Corps  Systems  Command  (MARCORSYSCOM)  provides  guidance 
regarding  the  relative  importance  of  the  capabilities  this  VUAV  could  possess,  such  as 
speed,  endurance,  and  sensor  performance. 

The  Marine  Corps  Warfighting  Lab  (MCWL)  has  been  tasked  with  answering  a 
vague  question.  Headquarters  Marine  Corps  (HQMC)  wants  to  know  “the  required  MEB 
ISR  capability  sets  needed  in  order  to  meet  the  following  2015  Expeditionary  Maneuver 
Warfare  requirements? 

1 .  Support  the  commander’s  rapid  planning  and  decision-making  process. 

2.  Maintain  a  comprehensive  ISR  network  to  support  multiple  concurrent 
expeditionary  operations. 

3.  Facilitate  operational  maneuver  and  precision  engagement.” 

(Commandant’s  Sub  Team  Guidance,  2003) 

Based  on  recent  performance  and  a  dynamic  retasking  capability,  the  UAV  has 
been  determined  to  be  the  key  component  of  the  sensor  network  required  to  meet  these 
requirements.  Detailed  analysis  is  required  to  detennine  the  capability  set  for  this  MEB 
commander’s  UAV  of  the  future. 


There  are  over  57  UAVs  in  development  or  production  by  corporations  in  the 
United  States  and  over  216  worldwide.  Each  has  its  own  unique  capabilities  and  design 
factors.  These  design  factors  combine  with  various  uncontrollable  factors  like 
geography,  terrain  and  enemy  capabilities  to  form  a  very  difficult  problem  when 
attempting  to  determine  the  most  important  factors  and  the  appropriate  needs  of  the 
Marine  Corps.  A  problem  that,  even  with  the  fastest  computers,  most  efficient 
simulations,  and  a  team  of  analysts,  would  take  more  than  a  life  time  to  answer  using 
traditional  experimental  designs.  A  smarter  design  is  required  to  comprehensively 
explore  how  these  factors  affect  a  UAV’s  ability  to  perform  in  expeditionary  operations. 

This  study  looks  at  UAV  operations  in  the  Sea  Viking  scenario  provided  by 
MCWL  in  the  MANA  agent-based  modeling  environment  utilizing  robust  design, 
Orthogonal  and  Nearly-orthogonal  Latin  hypercubes,  data  farming  techniques,  the  Maui 
High  Performance  Computing  Center,  and  the  JMP  Statistical  Discovery  Software 
package.  The  Sea  Viking  Fleet  Battle  Experiment,  the  Marine  Corps’  largest  annual 
experiment,  provides  a  credible  scenario  for  model  development.  The  model  uses 
stochastic  techniques  to  consider  the  effects  of  terrain,  the  enemy  portrayed  in  the  Sea 
Viking  scenario,  and  UAV  operations  from  the  MEB  on  the  Amphibious  Readiness 
Group  off  the  coast  of  Camp  Pendleton.  A  sample  screen  shot  of  a  typical  starting 
condition  is  shown  below. 
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Six  UAV  elements  are  explored:  routing,  time,  number  of  UAVs,  speed,  sweep 
width,  probability  of  classification,  and  employment  considerations.  Five  enemy 
capabilities  are  also  considered:  detection  range,  stealth,  time  critical  target  frequency 
and  duration,  and  relationship  with  neutrals  or  non-combatants.  In  all,  over  130,000 
mission  simulations  produce  the  measures  of  effectiveness:  proportion  of  enemy 
classified  per  hour  and  proportion  of  enemy  classified  per  mission. 

Classification  and  Regression  Trees  (CRTs)  provide  a  way  to  analyze  the 
relationship  between  factors  and  the  MOE.  A  regression  tree  is  a  recursive  partition  of 
the  raw  data  into  sets  of  inputs  containing  similar  responses.  Partitioning  of  the  data 
occurs  successively  according  to  the  optimal  splitting  value  determined  from  all  possible 
values  of  each  available  variable.  The  optimal  splitting  value  is  the  value  of  the  predictor 
variable  that  minimizes  sum  of  square  error  among  all  predictors.  After  each  split,  the 
next  optimal  split  is  determined  within  each  partition.  This  may  be  the  same  variable  as 
the  initial  split  or  a  different  variable  obtained  from  all  available  factors  and  can  be 
different  for  each  partition.  Considering  each  partition  conditionally  independently  of  the 
previous  partitions  automatically  accounts  for  interactions. 

The  CRT  on  the  following  page  is  a  recursive  partition  of  the  raw  data  from  all 
43,560  MANA  runs  on  all  controllable  and  uncontrollable  factors  for  the  one  UAV 
scenario.  As  partitioning  of  the  data  proceeds,  the  most  significant  factors  produce  the 
first  splits.  The  partitioning  point  for  a  factor  range  suggests  an  upper  or  lower  limit  for  a 
factor  capability  producing  significant  improvement  in  the  second  MOE:  proportion  of 
enemy  classified  per  mission.  Each  box  (or  node)  indicates  the  optimal  factor  to  partition 
upon  and  the  optimal  level  of  the  split  itself.  Details  within  the  box  include  the  number 
of  data  points  within  the  node,  the  mean  enemy  classification  proportion  per  mission,  and 
the  standard  deviation  within  the  node. 
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As  expected,  endurance,  referred  to  as  “max  time,”  is  the  primary  factor  when 
considering  the  total  amount  of  enemy  classified  during  a  mission.  A  UAV  on  station  at 
least  seven  hours  will  classify  nearly  twice  the  proportion  of  the  enemy  than  a  UAV  with 
fewer  than  seven  hours  on  station  time  when  averaged  over  all  the  other  variables. 
Additionally,  given  seven  hours  to  search,  a  UAV  with  any  sweep  width  or  low 
probability  of  classification  will  perform  reasonably  well.  This  suggests  that  endurance 
can  make  up  for  moderate  short  comings  in  sensor  capability. 

A  similar  analysis  on  the  proportion  of  enemy  classified  per  hour  reveals  that  the 
most  important  factor  when  considering  time  sensitive  Intelligence  Preparation  of  the 
Battlespace  in  the  Sea  Viking  Scenario  is  the  sweep  width  of  the  UAV.  In  general,  wider 
sweep  widths  yield  higher  expected  proportions  of  enemy  classified  each  hour — as  much 
as  twice  as  much.  This  is  qualified  by  the  assumption  that  the  sensor  package  can 
maintain  a  fairly  high  probability  of  classification  as  the  sweep  width  increases.  In  less 
than  seven  hours,  a  UAV/sensor  package  capable  of  producing  a  probability  of 
classification  of  at  least  0.4  over  a  4,500  meter  sweep  width  may  be  expected  to  produce 
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a  rate  of  enemy  classification  nearly  three  times  greater,  on  average,  than  a  UAV  that 
does  not  meet  these  standards  for  the  scenario  detailed  in  this  study.  This  may  be  crucial 
in  a  time-constrained  situation. 

Whether  considering  a  rate  of  classification  or  the  proportion  classified  for  an 
entire  mission,  use  of  tactical  routing  is  more  effective  than  traditional  search  patterns. 
This  makes  tactical  sense  and  lends  credibility  to  the  model.  For  employment 
considerations,  when  more  intelligence  is  available,  it  is  more  important  to  follow 
preplanned  routing  as  opposed  to  chasing  unclassified  contacts.  The  balance  between 
reactivity  and  strictly  following  a  route  is  difficult  to  quantify.  With  that  caveat, 
reserving  about  one-third  of  the  on  station  time  for  chasing  unknowns  and  using  the 
remainder  to  follow  a  tactical  route  appears  to  be  the  best  combination  for  the  Sea  Viking 
scenario.  A  large  sweep  width  and  low  probability  of  classification  may  result  in  too 
much  wasted  time  if  reactivity  is  high.  Conversely,  high  reactivity  can  be  effective  if  the 
sweep  width  is  low. 
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I.  INTRODUCTION 


A.  OVERVIEW 

The  United  States  Military  engages  in  conflicts,  peacekeeping  operations,  and 
power  projection  around  the  world.  As  the  world’s  greatest  superpower,  the  nation 
expects  ever-greater  achievements  worldwide  with  less  military  force  committed,  fewer 
American  casualties,  and  lower  costs — all  faster  than  ever  before.  While  no  force  may  be 
able  to  stand  up  to  the  U.S.  in  combat,  the  fog  of  war  often  creates  uncertainties  and 
difficulties  resulting  in  casualties.  The  military  attempts  to  lift  this  fog  of  war  and  reduce 
uncertainty  through  Intelligence,  Surveillance,  and  Reconnaissance  (ISR)  missions.  The 
more  knowledge  of  the  battle  space  our  commanders  have,  the  greater  their  ability  to  plan 
and  execute  a  successful  mission  with  minimal  losses. 

Unmanned  Aerial  Vehicles  (UAVs)  are  a  key  component  of  today’s  ISR  missions. 
UAVs  provide  intelligence,  a  dynamic  retasking  capability,  and  real-time  video  imagery. 
The  Intelligence  Officer  for  the  First  Marine  Expeditionary  Force  (I-MEF)  indicated  this 
combination  proved  to  be  invaluable  to  our  forces  on  the  ground  during  both  Operation 
Enduring  Freedom  (OEF)  and  Operation  Iraqi  Freedom  (OIF).  While  the  operators 
believe  current  UAV  assets  are  effectively  employed,  they  are  limited.  UAVs  are  in  great 
demand  and  only  the  highest  priority  missions  receive  their  support.  (Howcroft,  2003) 
As  a  result  of  Operation  Iraqi  Freedom  Major  Combat  Operations,  lessons  learned 
indicate  that  a  “better  asset  to  collect  battlespace  intelligence  is  crucial  to  the  way 
forward.”  (US  Joint  Forces  Command,  2004)  UAVs  provide  an  alternative  to 
complement  manned  aircraft  and  satellites  in  filling  this  gap. 

The  United  States  Marine  Corps  is  currently  reviewing  its  UAV  employment. 
During  OIF,  the  Marine  Expeditionary  Brigade  (MEB)  commander  utilized  two 
squadrons  of  Pioneer  UAVs.  Each  squadron  provided  the  capability  to  fly  one  UAV  for 
up  to  six  hours  with  a  maximum  range  of  over  170  nautical  miles.  The  Pioneer  can  be 
relieved  on  station  twice  with  current  assets  and  manpower,  achieving  a  total  coverage  of 
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18-hours  during  surge  operations  with  the  current  force  structure  in  each  squadron.  The 
Marine  Corps  desires  to  improve  this  capability  in  an  efficient  and  effective  manner. 
(Hirsch,  2003) 


B.  BACKGROUND  AND  MOTIVATION 

The  planned  Pioneer  replacement,  the  Vertical  Unmanned  Aerial  Vehicle 
(VUAV),  is  in  the  beginning  stages  of  conceptualization  and  currently  has  an  Initial 
Capabilities  Document  (ICD).  It  is  expected  to  proceed  through  the  design  and 
acquisition  process  for  fleet  use  in  2008.  (Headquarters  Marine  Corps,  2003)  Marine 
Corps  Systems  Command  (MARCORSYSCOM)  provides  guidance  regarding  the 
relative  importance  of  the  capabilities  this  VUAV  could  possess,  such  as  speed, 
endurance,  and  sensor  performance.  If  the  requirement  is  for  the  VUAV  to  fly  at  320 
knots,  and  one  design  achieves  only  310  knots,  should  it  be  ignored?  What  if  the  slower 
alternative  has  a  significantly  greater  endurance  or  sweep  width?  What  is  significant?  Is 
it  worth  the  added  cost  to  develop  a  VUAV  capable  of  400  knots  or  is  220  knots 
sufficient,  at  substantial  savings?  These  alternatives  require  analysis  because  intuition 
and  experience  alone  may  not  provide  the  best  answer. 

The  Marine  Corps  Warfighting  Lab  (MCWL)  has  been  tasked  with  answering  a 
vague  question.  Headquarters  Marine  Corps  wants  to  know  “the  required  MEB  ISR 
capability  sets  needed  in  order  to  meet  the  following  2015  Expeditionary  Maneuver 
Warfare  (EMW)  requirements? 

1 .  Support  the  commander’s  rapid  planning  and  decision-making  process. 

2.  Maintain  a  comprehensive  ISR  network  to  support  multiple  concurrent 
expeditionary  operations. 

3.  Facilitate  operational  maneuver  and  precision  engagement.” 

(Commandant’s  Sub  Team  Guidance,  2003) 

Based  on  recent  performance  and  a  dynamic  retasking  capability,  the  UAV  has 
been  determined  to  be  the  key  component  of  the  sensor  network  required  to  meet  these 
requirements.  Similar  to  the  questions  MARCORSYSCOM  has  regarding  the  VUAV, 
detailed  analysis  is  required  to  detennine  the  capability  set  for  this  MEB  commander’s 
UAV  of  the  future. 
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The  procedure  for  determining  the  future  capability  requirements  for  this  UAV 
involves  demanding  maximum  performance  based  on  expected  technology 
advancements.  This  is  not  only  speculative,  but  may  be  overkill  for  some  capabilities. 
For  example,  the  Marine  Corps  may  not  need  a  UAV  capable  of  72  hours  endurance  or 
500  knots  air  speed  for  the  250  nautical  mile  max  radius  expected  for  MEB  expeditionary 
operations.  (Hirsch,  2003)  There  is  no  formal  analysis  of  the  trade  space  provided  by  the 
various  capability  characteristics.  Currently,  once  the  requirements  have  been  set, 
meeting  them  is  a  pass  or  fail  situation.  That  is,  if  a  competing  contractor  proposes  a 
UAV,  the  product  either  meets  all  parameters  or  does  not.  If  a  UAV  under  consideration 
were  to  fly  2  knots  slower  than  required,  it  would  fail.  What  if  that  product  had 
endurance  10  times  greater  than  the  others?  MCWU  does  not  have  an  analysis  tool  to 
assist  them  in  detennining  the  value  of  one  capability  characteristic  over  another. 

There  are  over  57  UAVs  in  development  or  production  by  corporations  in  the 
United  States  and  over  216  worldwide.  The  UAVs  employed  by  the  US  military  today 
fly  at  speeds  between  40  and  340  knots,  with  a  combat  radius  from  one  nautical  mile  to 
an  over  the  horizon  capability.  Some  can  be  launched  by  hand,  some  from  a  ship,  and 
some  require  a  full  runway.  The  sensors  on  board  these  UAVs  have  various  sweep 
widths,  resolutions,  sampling  rates,  and  weather  effects.  (American  Institute  of 
Aeronautics  and  Astronautics,  2004) 

These  design  factors  combine  with  various  uncontrollable  factors  like  geography, 
terrain  and  enemy  capabilities  to  form  a  very  difficult  problem.  A  problem  that,  even 
with  the  fastest  computers,  most  efficient  simulations,  and  a  team  of  analysts,  would  take 
more  than  a  life  time  to  answer  using  traditional  experimental  designs.  A  smarter  design 
is  required. 

Each  year,  the  Navy/Marine  Corps  team  conducts  Fleet  Battle  Experiment  Sea 
Viking  in  Southern  California.  The  primary  objective  is  Command  and  Control  and  ISR 
development.  This  is  an  opportunity  to  validate  future  combat  systems  and  purposed 
implementation  concepts.  It  also  provides  a  scenario  for  basing  and  possibly  validating  a 
model.  (Marine  Corps  Warfighting  Lab  website,  2004)  The  model  scenario  for  this 
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study  adopts  that  from  Sea  Viking  2004,  making  the  results  applicable  for  developing 
employment  techniques  and  capabilities  to  be  evaluated  in  future  Sea  Viking 
experiments. 


C.  BENEFITS 

This  thesis  provides  benefit  in  five  main  areas.  First,  it  yields  insight  into  the 
relative  importance  of  various  UAV  capabilities  in  ISR  missions  for  expeditionary 
operations.  This  directly  assists  in  the  development  of  the  2008  Pioneer  replacement 
VUAV  currently  under  acquisition.  It  aids  in  determining  the  required  capabilities  of  a 
system  of  UAVs  to  meet  the  future  missions  of  the  MEB  commander  in  2015.  The 
research  provides  possible  mission  profiles  and  employment  techniques  for  validation 
during  future  Sea  Viking  Fleet  Battle  Experiments.  The  tactics  and  procedures  evolving 
from  modeling  supports  initial  Navy  doctrine  for  integrating  unmanned  vehicles  into 
maritime  missions  specifically  addressing  current  issues  from  the  Fleet.  Finally,  this 
thesis  provides  focus  for  future  analysis  involving  physics-based  simulations  such  as 
Combat  XXI  and  detennines  key  parameters  for  consideration. 


D.  THESIS  FLOW 

The  following  chapter  contains  a  discussion  of  Agent-Based  Models  (ABMs)  in 
Modeling  and  Simulation  (M&S)  and  the  ABM  used  for  this  thesis,  Map  Aware  Non- 
uniform  Automata  (MAN A).  The  scenario  upon  which  the  model  is  based  and  the 
representation  of  entities  in  the  model  is  also  presented.  Chapter  III  examines  the 
model’s  controllable  and  uncontrollable  factors  effecting  UAV  operations,  robust 
experimental  design  uses  for  this  work,  and  the  implementation  of  Orthogonal  and  Nearly 
Orthogonal  Latin  Hypercubes.  Chapter  IV  discusses  the  data  farming  process,  post 
processing  of  the  data  from  batch  runs,  and  data  analysis.  The  final  chapter  presents 
tactical  insights  and  suggests  some  possible  follow  on  work  for  future  Operations 
Research  theses. 
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II.  MODEL  DEVELOPMENT 


All  models  are  wrong ,  but  some  are  useful. 

-George  Box 


This  chapter  provides  a  brief  overview  of  Agent-Based  Models  (ABMs), 
discusses  how  their  use  contrasts  with  other  combat  models  available  today,  and 
introduces  the  ABM  Map  Aware  Non-uniform  Automata  or  MANA.  The  scenario 
implementation,  including  terrain  and  agent  portrayal  in  the  MANA  environment  based 
on  Fleet  Battle  Experiment  Sea  Viking  2004  is  discussed. 


A.  AGENT-BASED  MODELING 

In  today’s  world  of  high-priced,  high-tech  systems  with  competing  alternatives 
and  joint  considerations,  decision  makers  require  detailed  program  analysis.  Often  expert 
opinion  and  historical  references  do  not  provide  adequate  information  for  this  analysis. 
The  systems  may  even  be  too  new  to  have  “experts.”  Live  Test  and  Evaluation  (T&E), 
while  part  of  any  system  development,  is  often  too  costly  to  explore  the  full  range  of 
possibilities  that  warrant  consideration.  Furthermore,  T&E  tends  to  be  one  of  the  first 
areas  cut  when  time  or  money  is  tight.  (Hoivik,  2003)  This  is  where  modeling  and 
simulation  can  provide  perfonnance  expectations  and  insights.  The  Defense  Modeling 
and  Simulation  Office  (DMSO)  goal  is  to: 

Provide  readily  available,  operationally  valid  environments  for  use  by 
DoD  components: 

-  To  train  jointly,  develop  doctrine  and  tactics,  formulate  operational 
plans,  and  assess  warfighting  situations. 

-  To  support  technology  assessment,  system  upgrade,  prototype  and  full 
scale  development,  and  force  structuring. 

Furthermore,  common  use  of  these  environments  will  promote  a  closer 
interaction  between  the  operations  and  acquisition  communities  in 
carrying  out  their  respective  responsibilities.  To  allow  maximum  utility 
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and  flexibility,  these  modeling  and  simulation  environments  will  be 
constructed  from  affordable,  reusable  components  interoperating  through 
an  open  systems  architecture. 

(Defense  Modeling  and  Simulation  Office  website,  2004) 

Significant  problems  with  most  currently  used  combat  models  include  time  and 
manpower  requirements.  Building  a  data  base,  implementing  a  scenario,  completing  a 
statistically  sufficient  number  of  runs,  processing  the  output,  and  conducting  an  analysis 
often  requires  months.  Teams  of  specialists  develop  these  combat  models.  Often  the 
team  members  have  a  thorough  understanding  of  only  a  small  portion  of  the  model  that  is 
finally  produced.  Tying  all  the  pieces  together  can  prove  to  be  the  most  difficult 
developmental  piece.  The  Joint  Simulation  System  (JSIMS)  was  conceived  as  the 
Department  of  Defense’s  (DoD’s)  “Flagship”  simulation  to  model  joint  combat 
operations.  Despite  over  a  billion  dollars  spent  on  the  development  of  this  system, 
difficulties  in  integrating  the  many  programs  used  in  the  system  may  be  the  end  of  the 
initiative.  Many  in  the  field  believe  this  program  is  already  dead.  (Manago,  2004) 

It  is  not  uncommon  for  the  databases  developed  for  a  combat  model  to  have 
questionable  accuracy.  Sometimes  databases  may  hold  unclassified  parameter  values  for 
developmental  or  training  purposes.  Discovery  and  correction  of  errors  in  the  data  entry 
process,  whether  unintentional  mistakes  or  undocumented  temporary  guesses,  does  not 
always  occur  during  verification.  The  1999  DMSO  award  winning  simulation  Joint 
Conflict  And  Tactical  Simulation  (JCATS)  is  one  of  the  key  training  models  employed 
today.  During  a  training  exercise  it  was  discovered  that  the  sensor  for  a  Tomahawk  Land 
Attack  Missile  (TLAM),  a  long  range,  subsonic  cruise  missile,  was  in  fact  represented  by 
an  8x  magnification  binocular.  This  is  a  rather  serious  error.  (Manago,  2004) 

However,  this  is  not  an  attempt  to  question  current  M&S  efforts.  These  examples 
merely  bring  to  light  the  point  of  the  quote  at  the  beginning  of  this  chapter  by  George 
Box:  “All  models  are  wrong,  but  some  are  useful.”  In  the  end,  all  the  physics  equations 
and  detailed  parameters  of  high  resolution,  physics-based  models  feed  into  a  combined 
probability  or  weight.  This  probability  or  weight  feeds  into  another  complex  equation 
designed  to  detennine  a  hit,  detection,  movement,  or  other  outcome. 
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Another  approach  to  modeling  and  simulation  is  from  the  ground  up.  Rather  than 
attempting  to  simulate  activity  as  close  to  reality  as  possible  by  modeling  every  detail, 
why  not  model  only  those  entity  attributes  which  have  a  significant  impact  on  the 
situation?  ABMs,  also  called  “distillations,”  follow  this  philosophy.  (Marine  Corps 
Warfighting  Lab  website,  2004)  They  are  distillations  of  the  real  world.  Individual 
entities,  called  “agents,”  are  given  capabilities  and  behaviors.  Capabilities  may  include 
parameters  such  as  movement  speed,  available  weapon  systems  and  sensor  attributes. 
Behaviors  include  factors  like  an  agent’s  propensity  to  follow  orders,  congregate  with 
friendly  agents  or  attack  enemies.  Each  agent  is  autonomous  and  reacts  according  to  its 
behavior  characteristics  and  what  it  perceives  within  its  own  situational  awareness 
picture.  The  interactions  of  the  agents  on  the  digital  “battlefield”  result  in  scenarios 
which  resemble  key  characteristics  of  the  real  world  in  which  we  live. 

Most  ABMs  are  stochastic  with  each  iteration  of  a  scenario  resulting  in  a 
different  outcome.  (Lucas,  2003)  Execution  of  many  iterations  produces  expected 
outcomes  and  identifies  variations.  Uncommon  results,  or  outliers,  can  be  the  most 
interesting  cases.  Combining  this  with  the  ABM’s  easy  set  up  and  modification 
characteristics,  quick  run  time,  and  high  performance  computers  allows  for  exploration  of 
a  wide  range  of  parameter  values.  This  results  in  a  powerful  tool  for  an  operations 
analyst’s  toolbox. 

Data  farming  is  a  technique  commonly  used  in  conjunction  with  ABMs.  The 
purpose  of  data  farming  is  to  explore  the  effects  of  a  parameter  in  a  model  across  its 
range  of  possible  values.  Changing  a  parameter  and  viewing  its  effects  through  multiple 
iterations  of  a  simulation  enables  effective  estimation  of  the  impact  that  parameter  has. 
Completing  this  process  for  all  parameters  of  interest  ascertains  the  significant 
parameters.  (Brandstein  and  Horne,  1998) 

Project  Albert  is  a  division  of  the  MCWL  which  utilizes  data  fanning  and  ABMs 

as  a 

method  to  address  decision-maker’s  questions  that  applies  high  perfonnance 

computing  to  modeling  in  order  to  examine  and  understand  the  landscape  of 
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potential  simulated  outcomes,  enhance  intuition,  find  surprises  and  outliers,  and 
identify  potential  options.  (Marine  Corps  Warfighting  Lab  website,  2004) 

One  of  the  many  ABMs  in  the  Project  Albert  suite  is  Map  Aware  Non-Uniform 

Automata. 


B.  THE  AGENT-BASED  MODEL  MANA 

Map  Aware  Non-unifonn  Automata,  or  MANA,  is  the  agent-based  modeling 
environment  selected  for  this  research.  David  Galligan  and  Michael  Lauren  began 
development  of  MANA  for  the  New  Zealand  Army  and  Defense  Force  in  1999.  Contact 
information  is  provided  on  the  opening  MANA  screen  (Figure  1)  providing  tribute  to  the 
work.  Since  then,  MANA  has  been  added  to  the  data  fanning  tools  in  the  Project  Albert 
suite  of  ABMs  maintained  by  MCWL. 


** 

MANA  3  _ OTA ** 

DEFENCE  TECHNOLOGY  AGENCY 

Map  Aware  Non-uniform  Automata 

This  software  has  been  developed  by  David  Galligan  and  Michael  Lauren 
for  the  New  Zealand  Army  and  Defence  Force. 

Terms  of  use: 

This  software  belongs  to  the  The  Defence  Technology  Agency  (DTA),  New 
Zealand  Defence  Force.  It  is  not  to  be  distributed  without  the  permission  of  DTA. 


For  Scientific  Research  Purposes  Only 


BETA  VERSION 

Enquiries  regarding  this  software  should  be  directed  to  DTA  through  contact  points: 
David  Galligan  d.galligan@dta.mil.nz 
Michael  Lauren  m. Iauren@dta.mil. nz 


I  understand  I  can  only  use  this  software  subject  to  the  conditions  stated  above. 


Version  3.0.29 


piC!  J  P6C 


Figure  1.  Map  Aware  Non-uniform  Automata  opening  screen  provides  contact 

information. 
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In  the  MANA  environment,  the  agents  are: 

>  Map  Aware — Agent’s  situational  awareness  includes  the  depicted  terrain  as 
well  as  battlespace  activities  in  the  simulation. 

>  Non-uniform — Each  agent  may  have  different  behavior  parameters  and 
capabilities.  That  is,  they  do  not  all  have  to  move  or  act  the  same  way. 

>  Automata — Agents  react  independently  on  the  battlefield  according  to  their 
own  individual  situational  awareness  and  personalities. 

This  distillation  allows  for  graphical  depiction  of  the  terrain  and  agents  to  the 
desired  level  of  detail.  MANA  is  a  straight  forward  application  that  is  intuitive  and  easy 
to  use  with  a  well  developed  Graphical  User  Interface  (GUI).  The  data  farming 
techniques  built  in  provide  the  ability  to  explore  an  extensive  range  of  capabilities  in 
minimal  time.  More  details  are  readily  available  in  the  MANA  User’s  Manual. 
(Galligan,  2003) 

Developing  the  terrain  for  a  scenario  is  a  fairly  easy  process  making  use  of  the 
graphical  editor  built  into  MANA  which  produces  basic  scenario  maps.  If  a  scenario  map 
is  available,  it  can  be  converted  digitally  with  any  graphical  editing  software,  such  as  MS 
Paint,  Paint  Shop  Pro,  or  even  Power  Point,  into  a  fonnat  MANA  recognizes.  In  this 
ABM,  terrain  characteristics  affect  an  agent’s  movement  speed,  cover,  concealment,  and 
line  of  sight.  These  settings  are  relative  to  each  other  and/or  conversions  from  tactical 
parameters. 

Agent  parameters  in  MANA  are  in  four  basic  types:  personality  weightings,  move 
constraints,  basic  capabilities,  and  movement  characteristics.  The  personality  weightings 
determine  an  agent’s  propensity  to  move  toward  or  away  from  battlefield  entities  such  as 
waypoints,  cover,  concealment,  and  other  agents. 

Move  constraints  are  conditional  modifiers  to  the  personality  weights.  For 
example,  a  UAV  agent  may  be  more  likely  to  seek  out  and  follow  enemy  agents  when  it 
is  within  a  certain  distance  of  the  objective.  MANA  enables  use  of  real  world 
performance  parameters  for  basic  capabilities  for  conversion  to  the  scale  of  the 
simulation  scenario.  Basic  capabilities  include  parameters,  set  in  the  real  world,  that  are 
converted  to  the  scale  of  the  scenario.  These  include  maximum  speed,  sensor  range  and 
ability,  and  weapon  range  and  effect,  among  others.  Finally,  movement  characteristic 
parameters  determine  the  type  of  algorithm  used  and  the  degree  of  randomness  employed 
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by  an  agent.  This  affects  behaviors  like  obstacle  avoidance  and  the  effects  of  terrain. 
Over  200  parameters  may  be  employed  in  49  possible  states  for  each  agent,  providing  the 
ability  to  capture  a  broad  range  of  behaviors  and  capabilities  of  battlefield  entities. 
(Galligan,  2003) 

The  personality  page  from  the  Edit  Squad  Properties  menu,  displayed  in  Figure  2, 
shows  how  an  agent’s  propensities  for  movement  may  be  modified  from  -100  to  100  and 
given  an  effective  range.  In  this  case,  the  agent’s  propensity  to  move  toward  the  next 
waypoint  is  20,  indicating  a  desire  to  move  toward  it.  A  negative  propensity  value 
indicates  a  propensity  to  move  away,  such  as  this  example  shows  for  the  top  category, 
enemies.  Additionally,  MANA  allows  for  determining  the  effective  range  of  a 
personality  weighting.  In  this  case,  if  a  way  point  is  between  0  and  1000  cells  away,  the 
indicated  weighting  applies.  This  agent  also  has  its  greatest  propensity  to  move  toward 
cover.  The  behavior  for  this  agent  may  be  summarized  as  desiring  to  stay  covered  or 
protected  while  moving  towards  its  next  waypoint  and  staying  away  from  enemy 
contacts.  The  ratios  created  from  these  weightings  feed  the  movement  equation  along 
with  other  parameters  to  detennine  the  agent’s  most  desirable  move.  A  detailed 
description  of  how  the  movement  propensities  affect  the  agent’s  movement  can  be  found 
in  MANA’s  users  guide.  (Galligan,  2003) 
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Figure  2.  Example  of  MANA  Edit  Squad  Properties  page,  tab  one  of  nine: 
Personality.  The  display  shows  some  of  the  many  attributes  which  affect  an  agent’s 
behavior  on  the  battlefield  in  the  MANA  environment. 


The  support  for  ABMs  in  the  Project  Albert  suite  is  excellent.  It  ranges  from 
analysis  to  modeling,  and  provides  the  capability  to  make  large  batch  runs  remotely  via 
Internet.  Users  can  farm  hundreds  of  thousands  of  data  points  from  their  own  worksite. 
Scenario  development  support  for  MANA  users  is  available  from  the  software  developers 
themselves.  It  is  not  uncommon  for  the  programmers  to  provide  version  updates  within  a 
month  of  emerging  requirements  to  capture  a  key  element  or  vary  a  parameter  not 
available  in  the  current  version  at  a  user’s  request. 

During  the  time  frame  of  this  study  alone,  many  updates  and  versions  to  the 
MANA  program  have  been  released.  This  study  uses  only  version  3.0.29  because  of  its 
methodology  for  computing  line  of  sight  (LOS).  Subsequent  versions  calculate  (LOS)  in 
an  undesirable  manner  for  this  study.  If  a  line  between  observer  and  target  crosses  an 
obstruction  square  (i.e.,  with  0.92  concealment  in  MANA)  then  line  of  sight  is  not 
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possible.  To  determine  LOS,  version  3.0.29  calculates  the  probability  of  seeing  through 
each  square  between  the  observer  and  a  target  in  range.  Newer  versions  compute  this 
based  on  relative  elevations. 

Figure  3  is  a  visual  representation  of  the  difference  between  the  two  methods.  On 
the  left  is  the  LOS  depiction  for  an  agent  in  “Dense  Brush,”  “Light  Brush,  and  “Billiard 
Table,”  or  an  open  area,  for  the  version  used  in  this  study.  On  the  right  is  the  LOS 
depiction  for  the  same  agent  in  the  same  environments  for  subsequent  versions.  The 
table  in  the  middle  displays  the  values  for  Going,  Cover  and  Concealment  for  various 
terrain  types.  Notice  the  various  levels  of  Concealment  offered  by  the  Billiard  Table, 
0.00  in  black,  Light  Brush,  0.30  in  light  green,  and  Dense  Brush,  0.90  in  dark  green. 
Intuitively,  an  agent  should  see  less  in  Dense  Brush  than  in  Light  Brush  or  the  Billiard 
Table  terrain.  This  characteristic  is  only  demonstrated  in  version  3.0.29. 


Figure  3.  Line  of  sight  determination  in  version  for  this  study  versus  subsequent 

versions.  We  see  the  desirable  LOS  characteristics  of  version  3.0.29  on  the  left, 
versus  subsequent  versions  on  the  right.  [Best  viewed  in  color] 


C.  SEA  VIKING  SCENARIO 

The  Sea  Viking  Fleet  Battle  Experiment,  developed  by  MCWL,  provides  the 
scenario  for  the  model  to  explore  important  UAV  capability  questions.  Sea  Viking  is  an 
experimentation  program  designed  to  allow  exploration  of  ways  in  which  the  MAGTF 
can  be  transformed  to  increase  combat  power,  operational  versatility,  utility,  and 
deployability.  It  is  a  Navy/Marine  Corps  exercise  for  developing  C4ISR  techniques  and 
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tactics  that  improve  our  ability  to  focus  forward  presence  and  Operational  Maneuver 
From  The  Sea  (OMFTS).  The  desired  end-state  is  to  make  the  Ship-To-Objective 
Maneuver  (STOM)  concept  an  operational  reality  for  the  Navy/Marine  Corps  Team. 
(Marine  Corps  Warfighting  Lab  Website,  2004) 

The  annual  experiments  involve  imposing  a  reality-based  threat  scenario  from  a 
current  area  of  interest  on  operations  in  the  Southern  California  area.  Utilizing  Naval 
Base  San  Diego,  Marine  Corps  Air  Station  (MCAS)  Miramar,  Marine  Corps  Base  (MCB) 
Camp  Pendleton,  March  Air  Reserve  Base,  MCAS  Yuma,  and  MCB  29  Palms,  the  full 
range  of  tactical  operations  may  be  conducted  with  Marines  and  sailors  from  the  ship  to 
the  objective  via  air,  land,  and  sea. 

The  scenario  from  Sea  Viking  2004  provides  a  validated  threat  and  mission 
context  for  the  model  which  is  acceptable  to  the  principal  stakeholders  in  UAV 
development  for  the  Marine  Corps.  Mountain  Infantry  units,  Coastal  Infantry  units,  an 
Annor  unit,  and  Time  Critical  Targets  (TCT)  are  included  in  the  force  structure.  The 
host  nation  is  expected  to  provide  no  support.  The  geographic  region  is  approximately 
150  nautical  miles  by  150  nautical  miles.  Details  from  the  Sea  Viking  Scenario  are  in 
Appendix  A.  The  focus  for  this  study  is  on  the  Intelligence  Preparation  of  the 
Battlespace  (IPB)  during  the  initial  phases  of  operations  for  the  MEB.  This  directly 
addresses  the  first  portion  of  the  EMW  requirements  posed  to  MCWL  above,  as  well  as 
many  of  the  questions  from  the  VUAV  Project  Manager.  (Marine  Corps  Warfighting 
Lab,  2003) 

D.  INSTANTIATING  A  SEA  VIKING  SCENARIO  IN  MANA 

This  section  presents  the  modeling  concepts  and  includes  some  examples  where 
appropriate  to  provide  an  understanding  of  the  level  of  detail  in  this  study.  This  includes 
a  description  of  the  battlespace,  agent  development,  aggregation,  and  creative 
alternatives.  More  information  on  MANA  functionality  and  algorithms  may  be  found  in 
the  MANA  User’s  Manual.  The  details  of  the  model  development  and  the  final  model 
are  available  by  contacting  the  author  or  advisors. 
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The  model  includes  representations  of  the  Southern  California  terrain.  Sea  Viking 
enemy  organization  and  capabilities,  as  well  as  civilian  (neutral)  presence.  Friendly  force 
representation  is  limited  to  UAV  operations  in  support  of  the  ISR  mission.  Figure  4 
displays  a  screen  shot  from  the  start  of  a  typical  run.  The  numbered  circles  identify  the 
different  friendly  and  enemy  units  modeled. 

For  orientation  purposes,  the  display  shows  Los  Angeles  and  San  Diego  city  areas 
in  dark  yellow  and  29  Palms  (the  objective)  in  the  upper  right  corner.  Bright  yellow 
depicts  all  major  and  most  secondary  roads.  Over  515  agents  make  up  the  scenario, 
including  345  civilians,  9  enemy  tanks,  150  enemy  infantry,  10  enemy  TCTs,  and  1  to  3 
friendly  UAVs. 

Area  1  includes  the  UAVs,  two  in  this  case,  at  sea  on  Naval  ships.  Each  iteration 
of  the  scenario  starts  with  a  different  random  number  seed  which  varies  the  initial 
position  of  all  agents  on  the  battlefield.  The  UAVs  have  a  planned  tactical  route  to 
follow  over  land  to  cover  key  tactical  objectives.  Area  2  encompasses  the  Red  (enemy) 
Coastal  Infantry  agents  from  the  Sea  Viking  scenario  intended  to  patrol  the  beach  line. 
Area  3  identifies  the  Red  Low  Infantry  which  operate  in  low  level  terrain  outside  the  city. 
The  mission  for  this  group  is  to  move  towards  the  city  in  search  of  neutrals  to  convert  to 
enemy.  The  Red  Mountain  Infantry  are  in  area  4.  The  mountain  infantry  move  toward 
the  choke  point  at  the  juncture  of  Banning  Pass  and  Yucca  Valley.  Area  5  to  the 
Northeast  includes  the  Red  Objective  Area  Forces.  Infantry  on  the  objective  are 
protecting  the  senior  leadership.  Some  of  their  forces  move  down  Yucca  Valley  to  meet 
up  with  the  forces  coming  down  from  the  mountains.  In  the  Northwest  is  the  Red  Armor 
Unit  within  area  6.  The  armor  moves  southeast  to  take  a  defensive  position  in  the 
vicinity  of  Banning  Pass.  Center  stage  in  area  7  are  the  TCTs  located  in  the  channelized 
terrain  representing  a  Surface-to-  Air  Missile  (SAM)  systems  or  Anti-Aircraft  Artillery 
(AAA).  Throughout  the  region  are  yellow  “Neutral”  or  civilian  agents. 
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Figure  4.  MANA  Sea  Viking  base  scenario.  The  display  shows  a  sample  initial 
friendly,  enemy  and  civilian  agent  layout  within  the  battle  space.  [Best  viewed  in 

color] 

Key  enemy  characteristics  include  movement  speed,  dispersion,  tendency  to  seek 
cover  &  concealment,  sensor  range,  enemy-neutral  cohesion,  stealth  or  detectability 
characteristics,  and  a  mission.  UAV  parameters  include  speed,  endurance,  sweep  width, 
sensor  capability,  reactivity  and  number  of  units  employed. 

1.  The  Battlespace 

As  mentioned,  terrain  depiction  in  MANA  is  straight  forward.  Multiple  JOGAIR 
maps  of  the  Southern  California  Area  make  up  the  official  Sea  Viking  2004  Scenario 
map.  (Marine  Corps  Warfighting  Lab,  2003)  The  area  of  operations  is  identified, 
captured  digitally,  and  enhanced,  tracing  over  roads,  urban  terrain,  water,  desert,  and  light 
and  dense  vegetation  areas  in  visually  appropriate  colors,  with  specified  RGB  values. 
After  being  converted  to  bitmap  format,  the  image  is  imported  to  MANA.  Using  the 
MANA  Scenario  Map  Editor,  the  values  for  the  parameters  that  determine  how  these 
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different  terrain  features  affect  an  agent’s  speed,  concealment,  and  cover  are  entered. 
Many  of  the  values  used  are  default  settings  in  MANA.  Others  result  from  previous 
MANA  work,  interpolation,  and  the  tactical  experience  of  several  Marine  officers. 
Figure  5  depicts  the  values  from  the  MANA  Scenario  Map  Editor. 
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Figure  5.  Terrain  effects  parameter  values  as  displayed  in  the  MANA  Scenario  Map 

Editor. 


Terrain  parameter  values  for  Road,  Light  Brush,  and  Dense  Brush  utilize  the 

default  settings,  make  intuitive  sense,  and  have  been  utilized  previously  in  other  studies. 

(Brown,  2000)  For  example,  the  terrain  type  “Road”  has  a  parameter  value  of  1.00  for 

“Going,”  meaning  an  agent’s  speed  parameter  value  is  multiplied  by  a  factor  of  1.00,  or 

unaffected,  since  roads  are  not  intended  to  limit  movement.  Conversely,  the  “Cover”  and 

“Conceal”  parameters  for  the  “Road”  terrain  type  are  both  zero  since  roads  provide  no 

cover  or  concealment.  Terrain  types  that  provide  cover  have  a  value  greater  than  zero  but 

less  than  one,  indicating  the  probabilistic  effect  the  terrain  has  on  the  likelihood  that  an 
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agent  in  that  terrain  would  be  seen  by  another  agent  or  hit  if  shot  at.  Concealment 
providing  terrain  only  affects  an  agent’s  probability  of  being  seen.  The  parameter  values 
for  City,  Desert,  and  Water  terrain  types  were  developed  for  this  study.  They  also  make 
intuitive  sense  and  were  reviewed  with  sample  scenarios  in  the  MANA  environment  to 
ensure  an  effective  representation. 

MANA  displays  maps  from  bitmap  files  in  a  maximum  resolution  of  1000  by 
1000  cells.  This  setting  results  in  each  cell  equating  to  about  295  meters  on  a  side  for  the 
area  of  operations  represented.  Each  time  step  in  MANA  equates  to  36  seconds  in  the 
real  world.  A  spread  sheet  provides  an  effective  means  to  compute,  display,  and 
reference  many  desired  conversions.  Figure  6  displays  details  for  this  scenario  using 
Microsoft  Excel  for  computing  the  bounding  corners  of  the  scenario  in  latitude  and 
longitude,  conversions  to  nautical  miles,  statute  miles,  kilometers,  meters,  feet  and 
MANA  cells.  This  information  is  used  in  developing  movement  and  sensor  capabilities 
for  the  agents  in  the  MANA  model. 
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Due  to  the  size  of  the  geographic  area  in  the  scenario,  consideration  is  given  to  the 
variation  in  distance  between  lines  of  longitude.  At  the  latitude  for  the  center  of  the  Sea 
Viking  Fleet  Battle  Experiment,  one  degree  of  longitude  is  equivalent  to  about  49.83 
nautical  miles  vice  60  nautical  miles  at  the  equator.  Figure  6  shows  the  effect  of  this 
detail  in  the  MANA  battlefield. 

2.  Agent  Development 

The  values  for  the  four  types  of  agent  parameters,  mentioned  previously,  are 
developed  by  a  variety  of  means.  Some  parameters,  such  as  agent  movement  speed,  can 
be  easily  detennined  based  on  known  real  world  values  converted  to  the  scale  of  the 
model.  For  example,  a  tank  which  travels  at  a  maximum  speed  of  60  miles  per  hour  in 
the  real  world  will  move  1.09  cells  per  time  step  in  this  MANA  scenario.  Others  are 
more  difficult  to  determine,  such  as  a  UAV’s  propensity  to  move  towards  the  next  way 
point.  Difficulties  arise  when  attempting  to  detennine  the  value  to  use  for  that  same 
UAV’s  propensity  to  move  toward,  or  follow,  detected  enemy.  What  should  be  the  ratio 
between  these  two  competing  goals? 

This  is  where  data  farming  comes  in.  Farming  is  the  act  of  running  the  simulation 
for  multiple  iterations  at  a  variety  of  levels  to  determine  the  effect  of  that  parameter  on 
the  scenario  outcome.  These  results  provide  appropriate  values  to  set  for  questionable 
parameters  to  produce  a  behavior  that  makes  logical  and  tactical  sense.  Variations  of 
significant  factors  in  the  final  analysis  help  to  focus  on  the  effects  of  these  factors  and 
their  interactions  with  other  variables  in  the  model. 

The  primary  focus  of  this  study  is  the  ability  of  a  UAV  to  detect  and  classify 
forces  on  the  ground.  To  this  end,  priority  is  on  modeling  UAV  sensor  capabilities  as 
well  as  its  endurance,  speed,  routing,  and  tactical  employment.  Any  particular  sensor  has 
a  specified  Field  of  View  (FOV).  The  sweep  width  of  the  UAV/sensor  combination 
changes  with  altitude.  For  example,  a  15°  FOV  sensor  employed  at  2000  feet  Above 
Ground  Level  (AGL)  has  a  sweep  width  of  about  527  feet.  This  same  sensor  employed  at 
14,000  feet  AGL  will  have  a  sweep  width  of  about  3686  feet.  The  predominate  factor  in 
this  situation  is  sweep  width.  With  this  in  mind,  sweep  width  is  modeled  for  this  study 
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with  the  knowledge  that  a  FOV/flight  altitude  combination  may  be  derived  for  a  given 
sweep  width. 

Figure  7  pennits  determination  of  the  sweep  width  for  a  given  combination  of 
FOV  and  flight  altitude.  The  plots  show  how  quickly  a  10,000  meter  FOV  is  achieved  at 
a  moderate  altitude  considering  that  potential  design  altitudes  for  UAVs  are  as  high  as 
40,000  feet  AGL.  The  vertical  axis  provides  the  sweep  width  in  meters.  The  horizontal 
axis  lists  altitudes  up  to  25,000  feet  AGL.  Each  line  represents  a  different  FOV.  Thus, 
for  a  particular  FOV,  the  left  axis  indicates  the  sweep  width  obtained  at  a  given  altitude. 
Sweep  width  is  varied  from  2,000  meters  to  10,000  meters  in  this  study  to  capture 
significant  possibilities. 


Sweep  Width 


FOV  15 

— ■— 

FOV  30 

FOV  45 

FOV  60 

Altitude  (Feet  AGL) 


Figure  7.  Graph  of  sweep  width  obtained  given  a  flight  altitude  and  FOV.  The 
display  shows  the  range  of  possible  sweep  width  values. 


The  other  sensor  factor  which  deserves  attention  is  probability  of 

detection/classification.  For  a  given  sensor,  the  single  glimpse  probability  of 

detecting/classifying  a  given  target  at  a  given  range  is  generally  known — under  ideal 
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conditions.  Ideal  conditions  may  include  weather,  solar/lunar  position,  atmospheric 
conditions,  clutter,  contrast,  and  signal  strength.  For  example,  an  optical  sensor  my  have 
a  probability  of  detection  of  0.7  for  a  tank  from  10,000  feet  AGL  with  clear  skies,  a 
relative  humidity  of  less  than  20%,  and  the  sun  overhead.  As  cloud  cover  increases, 
humidity  increases,  the  sun  changes  position,  a  dust  storm  pops  up,  or  condensation 
builds  up  on  the  lens,  this  probability  will  drop.  Also,  higher  altitudes  are  associated  with 
reduced  resolution,  signal  strength,  increased  interference  as  well  as  other  confounding 
effects  on  probability  of  detection/classification. 

In  this  MANA  model,  single  glimpse  probability  is  the  probability  of  classifying  a 
target  in  one  time  step.  As  the  sensor  sweeps  an  area,  a  target  in  the  area  may  pass  down 
the  center  of  the  sensor  path  or  may  be  near  the  edge.  If  the  target  is  near  the  edge,  it 
may  only  be  possible  to  acquire  it  on  a  single  time  step.  In  this  case,  target  classification 
likelihood  is  the  single  glimpse  probability.  On  the  other  hand,  if  a  target  passes  down 
the  center  of  the  sensor’s  path,  it  may  be  in  range  for  several  time  steps.  In  this  case,  the 
probability  of  classifying  this  target  is  additive,  using  the  laws  of  probability  and 
assuming  independent  glimpses,  for  each  time  step  it  is  within  the  sweep  width. 

Figure  8  displays  a  sampling  of  the  various  resulting  lateral  range  curves  modeled 
in  this  study.  The  maximum  and  minimum  capability  UAVs,  in  regard  to  resulting 
probability  of  classification,  are  presented  along  with  three  intermediate  examples.  Each 
line  provides  the  probability  of  classifying  a  target  for  the  given  distance  from  the  UAV’s 
flight  path  indicated  along  the  x-axis.  The  most  capable  sensor  nearly  equates  to  a  cookie 
cutter  with  a  probability  of  detection  of  1  within  about  9,500  meters.  This  UAV  is 
traveling  at  the  slowest  setting,  100  knots,  with  the  widest  sweep  width,  10,000  meters, 
and  the  highest  probability  of  classification,  0.9,  for  a  single  glimpse. 
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Figure  8.  A  visual  sample  of  the  lateral  range  curves  modeled.  The  display  shows 
the  resulting  probability  of  classification  as  a  function  of  lateral  range  from  the  UAV 
for  a  sensor  with  various  probabilities  of  classification  at  various  airspeeds. 


In  this  study,  the  actual  single  glimpse  probability  assigned  to  the  sensor  varies, 
accounting  for  possibilities  beyond  that  which  exists  or  is  expected  in  the  near  future. 
This  demonstrates  another  advantage  of  the  data  farming  process:  the  ability  to  detennine 
if  a  significant  advantage  may  result  from  a  capability  outside  of  what  is  currently  under 
consideration.  Probability  of  detection  ranges  are  varied  from  0.1  to  0.9,  allowing  for 
consideration  of  more  highly  capable  sensors  than  are  currently  available  as  well  as 
degradation  to  current  sensor  capabilities,  due  to  weather,  altitude,  or  other  factors. 


3.  Aggregation 

Force  aggregation  is  a  common  technique  in  modeling  and  simulation.  The 
primary  model  used  for  this  study  does  not  have  an  agent  for  each  soldier  described  in  the 
Sea  Viking  scenario.  Approximately  one-third  the  number  of  infantry,  tanks,  and  TCTs 
are  portrayed  in  the  primary  (aggregated)  model  to  reduce  the  run  time  of  the  MANA 
software  package.  The  aggregated  model  runs  in  less  than  thirty  seconds  where  as  the 
full  model  can  take  over  four  and  a  half  minutes  on  a  2.81  GHz  Pentium  4  with  512  MB 
of  RAM.  This  makes  a  big  difference  in  the  time  required  to  complete  hundreds  of 

thousands  of  runs.  The  robust  design  implementation  (Section  3.3)  to  explore  the  desired 
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variety  of  factors  would  take  over  a  year  of  CPU  time  to  complete  with  the  full 
implementation  of  the  enemy  forces  in  the  Sea  Viking  scenario.  A  consideration  is 
whether  this  representation  of  only  one-third  the  forces  is  valid. 

A  validation  test  with  the  full  level  of  enemy  forces  run  many  times  at  high, 
medium,  and  low  levels  for  all  variables  can  determine  if  this  aggregation  scales 
properly.  These  levels  are  similarly  run  in  the  aggregated  model.  The  resulting  data 
provides  the  proportion  of  enemy  classified  in  both  the  full  model  and  the  aggregated 
model  under  equivalent  conditions  at  three  different  levels.  Analysis  of  this  data  using 
the  nonparametric  Kruskal- Wallis  Test  (Conover,  1999)  indicates  that  the  distribution  of 
detection  ratios  produced  from  each  of  these  models  are  statistically  indistinguishable. 

Figure  9  displays  plots  of  two  of  the  levels.  The  vertical  axis  provides  the 
proportion  of  enemy  classified  for  a  particular  run,  which  is  listed  on  the  horizontal  axis. 
The  graph  shows  the  similarity  between  the  non-aggregated  and  aggregated  models  at 
high  levels,  roughly  between  0.2  and  0.3,  and  the  low  levels,  below  0.05.  The  conclusion 
from  the  aggregated  and  full  comparison  is  that  analysis  based  on  the  aggregated  scenario 
is  valid  for  the  full  scenario  in  regard  to  classification  proportion. 


Figure  9.  Graph  of  classification  proportions  from  twenty  runs  of  the  full  and 
aggregated  scenarios  displaying  the  equivalence  of  the  resulting  distributions  for  high 
and  low  factor  levels.  [Best  viewed  in  color] 
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4.  Measure  of  Effectiveness  and  Creative  Modeling  Alternatives 

MANA  enables  access  to  a  variety  of  parameters,  including  over  200  available  for 
data  fanning,  and  Measures  of  Effectiveness  (MOEs)  which  can  be  evaluated  for  each 
simulation  run.  The  primary  MOE  for  this  study  is  proportion  of  enemy  detected  over 
time.  MANA  can  record  detection  data  for  each  agent  at  each  time  step.  While  this 
provides  high  resolution  on  the  MOE,  for  such  a  large  scenario,  the  output  files 
eventually  crash  the  operating  system  due  to  a  limit  on  the  number  of  subdirectories 
within  a  directory.  Utilizing  the  default  output  from  MANA  provides  a  more  efficient 
sampling  of  the  MOE  that  proves  sufficient.  The  default  output  provides  a  summary 
statistic,  number  of  agents  killed,  for  each  type  of  agent  following  each  scenario  run. 
Over  multiple  iterations  at  each  set  of  parameter  values,  or  design  points,  a  mean  and 
standard  deviation  for  the  number  killed  for  each  agent  type  is  obtained. 

This  study  focuses  on  ISR,  looking  at  UAV  classifications,  not  at  its  ability  to 
destroy  the  enemy.  This  is  where  the  creative  modeling  comes  in.  By  providing  the 
UAV  with  a  weapon  having  a  probability  of  hit  of  1.0  and  unlimited  rounds,  the  UAV 
“kills”  each  agent  it  detects  and  classifies.  Since  the  UAV  flight  path  in  this  scenario 
does  not  cover  the  same  terrain  more  than  once,  multiple  detections,  or  lack  thereof,  is  a 
minimal  concern. 

False  alanns  occur  when  a  UAV  attempts  to  classify  an  unknown  contact  and  it 
turns  out  to  be  a  neutral  vice  an  enemy.  The  MANA  output  contains  this  infonnation  in 
the  form  of  number  of  neutrals  classified.  The  expected  outcome  is  a  ratio  between 
neutral  classified  to  enemy  classified.  Since  we  have  a  two  to  one  ratio  of  neutral  to 
enemy  agents  that  appear  on  the  battlefield  this  ratio  would  be  about  two  to  one  on 
average. 
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III.  DESIGN  OF  EXPERIMENTS 


A  challenge  in  conducting  analysis  on  various  UAV  capabilities  is  the  large 
number  of  factors,  their  wide  range  of  levels,  and  their  interactions.  This  chapter  presents 
the  factors  in  two  sets,  controllable  and  uncontrollable.  Discussion  follows  regarding  the 
utilization  of  methods  to  effectively  explore  the  parameter  space,  robust  design  and 
orthogonal  Latin  hypercubes  (OLH).  The  intent  is  to  relay  the  value  and  capabilities  of 
each  technique  as  well  as  provide  an  understanding  of  their  employment.  References 
provide  a  more  thorough  understanding  of  how  to  utilize  robust  designs  or  OLHs. 

A.  CONTROLLABLE  FACTORS— UAV  CAPABILITIES 

The  controllable  factors  potentially  affecting  a  UAV’s  ability  to  detect  enemy 
units  and  vehicles  on  the  ground  not  only  have  a  large  amount  of  variability,  but  also 
have  significant  interactions.  It  may  seem  obvious  that  the  desired  case  is  a  wide  field  of 
view  (FOV)  and  high  resolution  yielding  a  high  probability  of  detection.  In  the  real 
world,  as  FOV  increases,  resolution  decreases  for  a  particular  sensor.  Similarly,  it  may 
appear  desirable  to  have  the  fastest  UAV  possible  to  cover  the  most  area.  However,  there 
are  two  confounding  factors  in  this  case.  First,  the  faster  the  UAV  travels,  the  less 
endurance  it  generally  has.  This  may  result  in  a  requirement  for  more  UAVs  to  maintain 
coverage  of  an  area.  Second,  the  faster  a  UAV  searches  an  area  the  less  time  is  spent  on 
any  one  location,  driving  detection  probabilities  down. 

Each  of  these  factors  should  be  considered  in  system  design  to  develop  a  UAV  to 
accomplish  a  desired  mission.  This  study  focuses  on  the  UAV’s  ability  to  detect  enemy 
forces  during  the  intelligence  preparation  of  the  battlefield  in  a  MEB-sized  operation. 
The  controllable  UAV  factors  determined  to  be  of  greatest  importance  through 
preliminary  analysis  and  discussions  with  the  sponsor  include: 

■  Routing 

■  Number  of  UAVs  employed 

■  Time  available 

■  Speed 

■  Sweep  width  (function  of  Altitude  and  FOV) 

■  Classification  probability 

■  Reactivity 
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The  effect  of  the  first  of  these  factors,  UAV  routing,  was  explored  in  some  preliminary 
work  conducted  during  the  Project  Albert  International  Workshop  8  (PAIW  8)  in 
Singapore.  The  remaining  factors  are  the  focal  point  of  this  modeling  and  analysis  effort. 

The  route  a  UAV  follows  on  any  given  search  mission  affects  the  number  of 
enemy  forces  detected  and  classified.  If  the  UAV  does  not  fly  over  any  enemy  locations, 
there  are  no  detections  or  classifications.  In  military  operations  today,  forces  often  have 
some  idea  of  where  the  enemy  is  likely  to  be,  through  satellite  imagery,  ground 
intelligence,  or  terrain  analysis  of  avenues  of  approach.  We  do  not  rely  on  random  or 
generic  search  pattern  techniques  for  this  type  of  employment.  Tactical  routing,  designed 
around  where  our  forces  are  going  and  where  enemy  forces  are  likely  to  be,  is  preferred. 

Preliminary  analysis  examined  various  routing  considerations.  At  PAIW  8,  a 
team  of  defense  analysts  from  a  variety  of  countries  developed  several  independent  routes 
for  the  Sea  Viking  scenario.  Some  developed  routes  having  no  knowledge  of  the  enemy 
locations,  while  others  possessed  knowledge  of  general  enemy  locations,  such  as,  “An 
enemy  armor  unit  is  expected  to  be  located  north  of  the  large  urban  area.”  Each  route 
required  starting  and  ending  at  the  ship  off  the  coast.  Of  the  sixteen  different  routes 
produced,  three  categories  emerged.  One  category  was  a  lawn  mower  type  search 
pattern,  another  uses  knowledge  of  likely  enemy  locations  to  set  the  UAV  waypoints,  and 
the  last  uses  tactical  routing. 

The  study  produced  four  final  routes,  three  from  the  categories  described  above 
and  one  previously  approved  as  likely  routing  by  MCWL.  As  an  example,  the  routing 
approved  by  MCWL  appears  in  Figure  10.  It  was  originally  developed  based  on  terrain 
analysis  and  location  of  the  objective  area.  While  there  would  be  different  waypoints  for 
any  UAV  commander  who  planned  this  mission,  this  one  is  as  likely  as  any  that  might  be 
planned.  It  includes  the  urban  areas  and  routes  most  likely  to  be  traversed  on  ingress, 
channeling  terrain,  key  defense  points,  and  the  objective  area  itself. 
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Figure  10.  An  example  of  routing  for  a  single  UAV  approved  by  MCWL. 


The  focus  on  routing  as  one  of  the  primary  variables  yielded  10,560  MAN  A  runs 
across  264  design  points  (combinations  of  input  factors).  Figure  1 1  shows  the  distribution 
of  the  outcomes  from  the  traditional  search  pattern  type  routing  and  the  tactical  routing. 
The  results  clearly  indicated  that  tactical  routing  was  superior  to  random  or  lawnmower 
type  search  patterns.  The  95%  confidence  intervals  for  the  two  subsets  do  not  overlap, 
and  the  tactical  routing  is  over  60%  better,  on  average,  for  this  scenario.  This  makes 
intuitive  sense  and  provides  support  of  the  model.  Based  on  these  outcomes,  the  current 
study  uses  the  tactical  routing  approved  by  MCWL  for  all  runs. 
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Traditional  Search  Pattern 


Tactical  Routing 


Figure  11.  Distributions  of  the  proportion  of  enemy  classified  form  Traditional 
Search  Patterns  versus  Tactical  Routing  of  the  UAV  in  the  SV  scenario.  Note  the 
tactical  routing  proves  to  be  63%  better  on  average. 


The  number  of  UAVs  employed  affects  how  much  area  can  be  covered  and  how 
long  it  takes  to  complete  the  search.  Interactions  with  speed  and  sweep  width  are 
intuitive,  but  must  be  quantified.  Are  two  UAVs  flying  at  150  knots  better  than  one 
flying  at  300  knots?  How  do  different  sweep  widths  affect  the  performance?  An  upper 
limit  of  three  UAVs  has  been  selected  for  this  study  based  on  discussions  with  the 
sponsors  and  the  Sea  Viking  scenario.  It  has  been  detennined  to  be  unlikely  that  the 
Marine  Corps  will  have  the  funding  or  personnel  to  employ  more  than  three  UAVs 
simultaneously  as  standard  operating  procedure  for  a  Marine  Expeditionary  Brigade 
(MEB)  size  Area  of  Responsibility  (AOR).  (Hirsch,  2003) 

Time  and  space  separate  multiple  UAVs  active  in  the  scenario.  This  follows 
general  tactical  airspace  control  rules  which  indicate  that  aircraft  commanders  should 
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plan  for  deconfliction  by  a  minimum  of  two  of  the  following  three  means:  time,  space,  or 
altitude.  (Department  of  Transportation,  2004)  This  has  the  added  benefit  of  ensuring 
there  are  no  multiple  detections  possible  in  the  scenario.  All  of  these  points  support  the 
modeling  and  data  collection  method  described  in  the  previous  chapter,  using  kills  to 
record  classifications. 

Time  in  the  AOR,  run  time,  in  conjunction  with  the  proportion  of  the  enemy 
classified  provides  a  measurement  of  the  expected  rate  of  proportion  of  enemy  classified 
over  time  for  a  UAV  with  a  particular  set  of  capabilities.  In  the  real  world,  a  commander 
may  send  out  a  UAV  to  search  an  area,  but  the  information  coming  back  to  the  command 
center  is  constantly  monitored  and,  at  a  minimum,  hourly  updates  are  reported.  In  the 
simulation,  each  iteration  stops  at  a  predetermined  time  or  when  the  UAV  arrives  back  at 
the  ship.  Collecting  the  statistics  on  number  and  type  of  enemy  detected  up  to  the  stop 
point  provides  a  single  data  point  for  that  set  of  capabilities.  Many  iterations  of  the 
scenario  are  run  for  each  set  of  capabilities,  yielding  sample  means  to  enable  examination 
of  the  relative  effectiveness  of  a  set  of  capabilities  over  time. 

Endurance  is  a  difficult  factor  to  model.  Developing  different  routing  for  each 
endurance  level  is  not  practical  for  this  study.  Instead,  estimating  the  effects  of  time 
using  the  methods  described  above  provide  insight  into  endurance.  Routes  are  varied 
from  one  enabling  a  single  slow  UAV  to  complete  a  route  to  a  set  of  three  routes  for  three 
fast  UAVs. 

It  should  be  noted  that  many  other  factors  affect  the  required  endurance  level  for  a 
MEB  sized  operation.  The  first  is  the  many  possible  variations  in  scenario  parameters 
regarding  distance  from  launch  site  to  target  area.  Second,  for  Intelligence  Preparation  of 
the  Battlespace  (IPB),  knowing  how  much  infonnation  can  be  expected  hour  by  hour  may 
be  just  as  important  as  mission  by  mission.  Finally,  these  UAVs  are  currently  intended  to 
relieve  each  other  on  station,  moving  the  significance  of  endurance  considerations  from 
the  focus  of  this  study  to  deck  cycles  and  launch  and  recovery  issues.  These  tactical 
considerations  certainly  warrant  attention.  Manned  aircraft  generally  have  priority  on  the 
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ship  and  the  physics  of  how  well  a  UAV  fits  into  the  deck  cycle  for  shipboard  operations 
may  be  a  larger  driving  factor.  For  this  study,  the  endurance  needed  to  cover  the  SV  area 
is  assumed. 

In  concert  with  the  Sea  Viking  2004  scenario,  the  UAV  starts  each  mission  from  a 
ship  just  off  the  coast.  Limiting  run  time  to  halt  each  iteration  at  a  specific  experimental 
point  or  back  at  the  ship  yields  the  desired  performance  measure,  number  of  enemy 
classified,  up  to  that  point. 

UAV  speed  is  a  representation  of  the  airspeed  the  UAV  flies  at  for  the  mission. 
There  is  no  variability  in  airspeed  during  a  single  iteration  of  the  scenario.  Airspeed  in 
the  MANA  environment  is  obtained  by  multiplying  the  desired  real  world  airspeed  by  the 
nautical  mile  conversion  factor,  6.2713,  which  converts  a  nautical  mile  to  a  MANA  cell 
width  for  the  geographic  area  represented.  This  number  is  then  divided  by  three  to  keep 
all  possible  airspeed  values  within  the  parameter  limits  of  MANA.  This  scaling  by  a 
factor  of  1/3  is  conducted  for  all  movement  speeds  in  the  scenario,  thereby  keeping  all 
speed  relations  equivalent. 

The  range  of  speeds  considered  is  from  current  UAV  capabilities  to  a  speed 
comparable  with  some  manned  aircraft.  The  low  end  is  100  knots  and  the  upper  end  is 
400  knots.  There  are  many  proponents  for  a  UAV  which  will  have  dash  speed  over  an 
Osprey,  which  has  a  maximum  airspeed  of  305  knots  (Global  Securities,  2004).  While 
dash  speed  is  not  specifically  considered  in  this  scenario,  the  impact  of  this  airspeed 
capability  is  considered  for  completeness. 

UAV  sweep  width  describes  how  wide  the  search  area  is  for  a  single  glimpse  by 
the  sensor.  Sweep  width  is  modeled  using  the  sensor  range  parameter  in  MANA. 
Specifically,  the  UAV  agent’s  detection  and  classification  ranges  are  varied  in  lock  step. 
A  UAV  agent  detects  a  target  as  a  function  of  the  UAV’s  detection  range,  line  of  sight, 
the  target’s  stealth,  and  available  concealment  on  the  terrain.  Once  detected,  a  target  is 
then  classified  based  on  the  UAV’s  probability  of  classification  parameter,  discussed 
below. 

Sweep  width  ranges  from  2,000  meters  to  10,000  meters  to  encompass  the  current 
UAV  capabilities  and  future  possibilities.  Employing  a  UAV  at  over  25,000  feet  AGL  is 
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a  possibility  for  the  MEB  commander  and  Figure  7  in  the  previous  chapter  displays  the 
possible  sweep  widths  obtained  for  various  FOV.  Although  there  are  several  UAVs  on 

the  market  today  which  operate  at  altitudes  over  40,000  feet  AGL,  the  sweep  width  range 
in  this  study  encompasses  current  expectations  for  the  type  and  size  UAV  under 
consideration  by  the  Marine  Corps. 

UAV  classification  probability  is  the  probability  that  a  detected  target  is  classified 
as  friendly,  enemy,  or  neutral  on  a  single  glimpse.  In  this  model,  the  single  glimpse 
probability  of  classification  is  held  constant  throughout  the  range  of  the  sensor  for  any 
single  iteration.  The  resulting  effect  as  the  UAV/sensor  combination  travels  over  the 
ground  is  that  agents  on  the  edge  of  the  sensor  range  have  a  lower  probability  of 
classification  than  targets  which  the  UAV  passes  directly  over  and  are  within  range  for  a 
longer  period  of  time.  This  provides  a  different  lateral  range  curve  for  each  sweep 
width/probability  combination.  The  probability  of  classification  range  in  this  study  is 
from  0.1  to  0.9,  effectively  capturing  the  full  range  of  possibilities. 

UAV  reactivity  is  a  term  to  describe  the  UAV’s  propensity  to  follow  unclassified 
agents  or  to  stay  on  its  route.  It  is  how  “reactive”  the  UAV  is  to  contacts.  The  intent  is  to 
explore  the  possible  employment  considerations  a  UAV  commander  may  have.  Given 
any  area  to  search,  the  UAV  commander  will  ideally  plan  a  route  to  cover  the  entire  area. 
The  question  is,  once  flying  the  route,  should  the  UAV  commander  send  the  UAV  after 
unclassified  contacts  in  an  attempt  to  determine  if  they  are  enemy,  or  should  he  follow 
the  planned  routing  to  ensure  complete  coverage? 

In  MANA,  UAV  reactivity  is  modeled  by  changing  the  UAV’s  propensity  to 
follow  detected  “unknown”  or  unclassified  agents  relative  to  its  propensity  to  move 
toward  the  next  way  point.  This  ratio  is  varied  from  one-third  as  desirous  to  move 
towards  unknowns  to  three  times  as  desirous  to  move  towards  unknowns  in  MANA 
personality  weights.  In  any  one  time  step,  the  UAV  agent  will  consider  this  ratio  along 
with  the  proximity  of  other  UAVs  and  contacts  already  classified  as  enemy  in 
determining  its  next  move. 

Figure  12  presents  the  basic  set  up  for  a  UAV  in  MANA.  The  key  factors, 
movement  speed,  classification  range,  and  classification  probability  are  set  at  a  middle 
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level  for  the  base  case  and  varied  in  the  experimental  design.  These  middle  factors 
permit  a  realistic  run  time  viewing  that  is  easy  to  understand  and  follow  visually  for 
debugging  purposes. 


Figure  12.  Sample  UAV  Range  tab  from  Edit  Squad  Properties  function  in  MANA. 
The  display  shows  basic  speed,  sensor  detection  range,  and  sensor  classification  range 

and  probability  functional  areas. 


B.  ROBUST  DESIGN  AND  NOISE  FACTORS 

Robust  design  was  pioneered  by  Genichi  Taguchi  in  the  1980s  for  quality 
planning  and  engineering  product  design  activities.  (Taguchi  and  Wu,  1980)  The  fact 
that  often  a  process  may  contain  many  variables  which  may  be  uncontrollable,  or  costly 
to  control,  can  weigh  heavily  on  the  best  course  of  action  to  take  in  attempts  to  optimize 
that  process.  For  example,  if  the  enemy  has  the  ability  to  detect  an  approaching  UAV 
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before  becoming  within  range  of  the  UAV’s  sensors,  the  enemy  may  hide  and  perhaps 
avoid  detection.  If  the  enemy’s  detection  range  were  known,  a  UAV  could  be 
constructed  which  has  a  sensor  range  greater  than  that  of  the  enemy.  However,  the 
enemy’s  detection  range  is  variable.  In  the  case  of  counter  detection,  it  can  depend  on 
wind  speed,  wind  direction,  flight  altitude,  and  background  noise.  Additionally,  the  cost 
of  increasing  the  sensor  range  of  a  UAV  sensor  is  generally  a  loss  in  resolution  of  that 
sensor,  monetary  costs  aside.  (Federation  of  American  Scientists,  2004)  This  loss  in 
resolution  may  be  too  great  a  cost  for  effective  classification  of  targets. 

Traditional  experimental  designs  attempt  to  hold  uncontrollable  variables,  or 
‘noise’  variables,  constant.  This  is  intended  to  ensure  that  the  impact  due  to  these 
uncontrollable  variables  is  constant  throughout  the  experimental  runs.  This  results  in  an 
“apples  to  apples”  comparison  and  some  mean  performance  indicator  or  Measure  of 
Effectiveness  (MOE)  may  be  obtained.  However,  this  also  results  in  decisions  made  on  a 
narrow  set  of  circumstances.  A  robust  design  ensures  the  controllable  factor  levels  are 
optimized  with  regard  to  the  uncontrollable  variables  that  affect  performance.  (Sanchez, 
1994)  By  exploring  the  influence  of  noise  variables,  a  set  of  design  parameters,  which 
may  not  perform  the  best  in  a  particular  instance,  may  perform  best  across  a  wide  variety 
of  circumstances  in  which  the  US  military  finds  itself  employed  these  days. 

The  intent  of  distillations  is  not  to  model  every  aspect  of  reality,  but  to  focus  on 
the  significant  factors  and  relationships.  (Brandstein  and  Home,  1998)  The  significant 
factors  can  be  effectively  determined  through  iterative  evaluation  of  the  noise  variables. 
The  end  result  is  a  subset  of  the  significant  factors  which  are  uncontrollable  in  the  real 
world  but  whose  effects  can  be  explored  through  M&S. 


C.  UNCONTROLLABLE  FACTORS— ENEMY  CAPABILITIES 

In  developing  this  model,  several  factors,  not  typically  under  the  control  of  the 
UAV  developer  or  MEB  commander,  require  investigation.  The  following  factors  were 
determined  to  be  significant  enough  to  warrant  inclusion  in  the  subset  of  uncontrollable 
factors  in  the  final  analysis: 

■  Enemy  detection  range 

■  Enemy  stealth 
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■  Neutral-Enemy  cohesion 

■  TCT  vulnerability  frequency 

■  TCT  vulnerability  duration 

Enemy  detection  range  is  fairly  self-explanatory  and  covered  in  a  previous 
example.  Enemy  stealth  is  equally  intuitive.  It  represents  how  well  a  target  proceeds 
unnoticed.  In  the  MANA  environment,  this  is  the  weighted  probability  that  an  agent  is 
not  seen  by  a  particular  agent  in  any  one  time  step  regardless  of  other  concealment, 
cover,  or  line  of  sight  factors.  These  two  factors  are  used  in  conjunction  with  a  MANA 
state  change.  When  the  enemy  detects  the  approaching  UAV,  it  conducts  a  state  change 
to  an  enemy  contact  state  where  the  agent  possesses  a  higher  stealth  to  simulate  a  duck 
and  cover  reaction. 

Enemy-Neutral  cohesion  represents  the  ability  of  combatants  to  congregate  with 
neutrals  in  an  attempt  to  avoid  detection.  The  MANA  parameter  controlling  this  aspect 
of  the  agent’s  behavior  is  the  propensity  to  move  towards  neutrals.  Along  with  the 
agent’s  desire  to  seek  concealment,  easily  traversed  terrain,  and  their  next  waypoint  while 
maintaining  dispersion,  this  is  a  weighted  value  in  the  movement  equation. 

Time  Critical  Target  vulnerability  frequency  is  a  measure  of  how  often  TCTs 
leave  their  hide  site  to  move  or  engage  targets,  for  example.  Similarly,  TCT  vulnerability 
duration  is  how  long  they  leave  their  hide  site  for  these  activities.  These  factors  are 
controlled  with  the  various  agent  state  changes  that  are  available  in  the  MANA  set  up.  In 
its  hide  site,  a  TCT  has  a  high  stealth  value  ensuring  that  the  probability  of  detection  is 
extremely  small.  When  the  TCT  is  out  of  its  hide  site,  it  is  vulnerable  to  detection. 

A  base  set  up  for  a  typical  enemy  infantry  squad  is  displayed  in  Figure  13.  The 
agent  is  given  a  negative  weight  for  “uninjured  friends”  to  provide  the  appropriate 
dispersion.  Positive  weights  for  other  factors  ensure  the  agent  moves  toward  its  intended 
objective  considering  concealment,  without  working  too  hard,  and  integrating  with 
neutrals. 
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Figure  13.  Red  Mountain  Infantry  squad  basic  personality  configuration.  Values  are 
manipulated  to  control  dispersion,  provide  a  mission,  effect  movement  propensities, 
and  manipulate  cohesion  with  neutrals. 


D.  ORTHOGONAL  LATIN  HYPERCUBES 

There  are  a  large  number  of  factors  worthy  of  consideration  between  the 
controllable  and  uncontrollable  factors.  A  problem  arises  in  attempting  to  effectively 
vary  these  factors  across  a  wide  range  of  possible  levels.  A  traditional  factorial 
experimental  design  tests  only  a  few  factors  at  two  or  perhaps  three  levels  each.  To 
utilize  this  approach,  some  factors  would  have  to  be  left  out  of  the  experimental  design 
and  linear  relationships  assumed.  A  smarter  design  is  required. 

An  Orthogonal  Latin  hypercube  (OLH)  design  is  chosen  for  its  excellent  space 
filling  properties,  the  resulting  low  correlation  between  factor  inputs,  and  ability  to 
identify  nonlinear  relationships.  (Cioppa,  2002)  OLHs  can  be  used  to  design  an 
experiment  evaluating  up  to  seven  factors  at  17  levels  each  in  an  efficient  and  effective 
manner.  Nearly  orthogonal  Latin  hypercubes  (NOLHs)  have  nearly  the  same  properties 
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with  slightly  higher,  but  negligible,  correlation  between  factors.  NOLHs  can  be  utilized 
to  evaluate  from  8  to  22  factors  at  up  to  129  levels. 

Figure  14  plots  each  design  point  derived  from  an  eight  factor  NOLH  with  33 
levels  for  each  factor.  This  NOLH  is  then  crossed  with  a  two-factor  factorial  design.  The 
factor  names  are  presented  down  the  diagonal.  Each  point  on  the  plot  represents  the 
corresponding  factor  levels  for  a  design  point.  The  NOLH’s  space  fdling  properties  are 
demonstrated  by  the  plots  in  contrast  to  the  factorial  design  factors.  Notice  the  lack  of 
space  fdling  represented  by  the  factorial  factors,  #UAVs  and  Routing,  compared  to  any 
of  the  other  factors  whose  design  points  were  obtained  from  a  NOLH  implementation. 
This  design  allows  for  the  exploration  of  many  variables  (in  this  preliminary  case,  ten) 
over  a  large  range  while  evaluating  many  points  within  the  range  for  each  appropriate 
variable.  This  ensures  the  ability  to  identify  nonlinear  relationships  and  interactions. 
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Figure  14.  Pairwise  scatter  plot  of  design  points  utilizing  a  nearly  orthogonal  Latin 
hypercube  crossed  with  a  factorial  design.  Factor  names  appear  along  the  diagonal. 
Each  dot  represents  a  design  point  for  the  corresponding  factors. 
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The  orthogonal  nature  of  the  design  results  in  no  significant  design-imposed 
correlation.  This  provides  the  ability  to  look  at  the  effects  of  each  variable  independently 
as  well  as  interactions  during  analysis.  Through  optimal  chaining  of  OLHs  or  NOLHs, 
the  space  filling  characteristics  can  be  increased  while  maintaining  the  orthogonal  nature 
of  the  design  and  no  significant  design  point  correlations.  (Cioppa,  2002)  This  provides 
a  greater  ability  to  analyze  multidimensional  data. 

This  study  uses  two,  optimally  appended,  orthogonal  Latin  hypercubes  for  each 
group  of  factors,  controlled  and  uncontrolled.  This  provides  33  design  points  for  each 
group  optimized  for  greatest  space  filling.  The  two  sets  of  33  design  points  each  are  then 
crossed  to  ensure  each  of  the  controllable  factor  design  points  are  evaluated  at  the 
maximal  range  of  possible  combinations  of  uncontrollable  factors  as  the  robust  design 
section  discusses  above.  This  results  in  33  times  33  =  1089  design  points  which  are  each 
run  for  the  three  factorial  cases  of  one,  two,  or  three  UAVs.  Finally,  these  3267  design 
points  are  each  run  for  40  iterations  to  take  advantage  of  the  stochastic  nature  of  MANA 
and  provide  a  look  into  the  variation  that  may  be  expected.  This  results  in  130,680  total 
MANA  runs  of  the  scenario.  Conducting  this  many  runs  with  a  typical,  physics-based 
combat  model,  such  as  JCATS  or  Combined  Anns  and  Support  Task  Force  Evaluation 
Model  (CASTFOREM)  would  take  years.  This  advanced  level  of  experimental  design 
with  typical  combat  models  is  undesirable  due  to  the  resources  required. 

The  ranges  for  the  controllable  and  uncontrollable  factors  in  both  the  real  world 
and  the  MANA  environment  are  displayed  in  Figure  15.  On  the  right  is  a  sample  of  the 
orthogonal  Latin  hypercube  for  the  controllable  factors  presenting  the  first  20  design 
points.  In  addition  to  the  40  replications  run  for  this  design,  a  separate  set  of  runs  of  5 
repetitions  each  is  available  to  be  used  as  a  test  set  for  verifying  the  models. 


37 


Factorial  Factors 

Units 

Re 

al 

MANA 

Orthogonal  Latin  Hypercube  CONTROLABLE  FACTORS 

1 

Number  of  UAVs 

UAVs 

1 

- 

i 

3 

1 

2 

3 

4 

5 

First  OLH  Factors  (Controllable) 

Low  Level 

2 

100 

2000 

0.1 

0.3 

1 

Time 

Hours 

2 

12 

* 

200 

1200 

High  Level 

12 

400 

10000 

0.9 

3.0 

2 

UAV  Speed 

Knots 

100 

400 

209 

836 

Time 

UAVSpeed 

sw 

ClassProb 

UAVReac 

3 

Sweep  Width 

Meters 

2000 

10000 

- 

7 

34 

Design  Point 

Time 

UAV  Speed 

Sweep  Width 

Classification  Prob 

UAV  Reactivity 

A 

Classification  Prob 

Prob. 

0.1 

0.9 

= 

1000 

9000 

1 

5.13 

400 

8564 

0.40 

1.0 

5 

UAV  Reactivity 

Ratio 

0.3 

3 

= 

10 

90 

2 

2.63 

175 

9155 

0.55 

0.3 

6 

1 

3 

3.25 

231 

2658 

0.30 

2.0 

7 

4 

3.88 

287 

4430 

0.90 

1.8 

Second  OLH  Factors  (Uncontrollable) 

5 

9.50 

381 

5611 

0.20 

1.2 

1 

Enemy  Det  Range 

Meters 

1000 

11000 

w 

3 

37 

6 

12.00 

194 

5020 

0.75 

0.5 

2 

Enemy  Stealth 

0 

100 

7 

8.25 

156 

10041 

0.35 

2.7 

3 

Neutral-Enemy  Cohesion 

-50 

50 

8 

7.63 

363 

7973 

0.85 

2.5 

4 

TCT  Frequency 

Hours 

2 

10 

200 

1000 

9 

7.00 

250 

6202 

0.50 

1.7 

5 

TCT  Availability 

Minutes 

20 

120 

33 

200 

10 

8.88 

100 

3544 

0.60 

2.3 

6 

11 

11.38 

325 

2953 

0.45 

3.0 

7 

12 

10.75 

269 

9450 

0.70 

1.3 

13 

10.13 

212 

7678 

0.10 

1.5 

14 

4.50 

119 

6497 

0.80 

2.2 

3 

DP  from  Factorial 

15 

2.00 

306 

7087 

0.25 

2.8 

33 

DP  from  frist  OLH 

16 

5.75 

343 

2067 

0.65 

0.7 

33 

DP  from  second  OLH 

17 

6.38 

137 

4134 

0.15 

0.8 

3267 

Total  Design  Points 

18 

4.50 

194 

6497 

0.75 

2.8 

40 

Runs  per  DP 

19 

2.00 

119 

7087 

0.80 

1.2 

130680 

Total  MANA  runs 

20 

8.25 

137 

10041 

0.15 

2.5 

Figure  15.  Design  of  Experiments  for  the  primary  analysis.  The  display  shows  the 
factors,  factor  ranges,  required  number  of  runs,  and  a  sample  of  the  design  points  and 

levels  for  the  controllable  factors. 


This  experimental  design  took  just  under  48  hours  to  run  at  the  Maui  High 
Performance  Computing  Center  (MHPCC).  Using  traditional  full  factorial  designs,  this 
analysis  would  require  over  1.8  x  10  runs.  This  would  not  finish  running  on  the  fastest 
computers  available  today  before  the  sun  bums  out.  A  graphical  comparison  is  provided 
in  Figure  16. 
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Figure  16.  Graphical  comparison  of  the  number  of  runs  required  to  conduct  this 
analysis  with  a  traditional  full  factorial  design  versus  a  smarter  design  with 
orthogonal  Latin  hypercubes.  [Best  viewed  in  color] 
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IV. 


DATA  ANALYSIS 


Orthogonal  Latin  Hypercubes  (OLHs)/Nearly-Orthogonal  Latin  Hypercubes 
(NOLHs)  and  MANA  output  facilitate  post  processing  and  data  analysis.  A  quick  review 
of  the  collection  and  preparation  of  data  for  analysis  begins  this  chapter.  Next  is  a 
discussion  of  the  statistical  modeling  techniques  utilized  to  gain  an  understanding  of  the 
relationship  between  the  Measure  of  Effectiveness  (MOE)  and  the  predictor  variables  and 
the  results  they  produce. 


A.  DATA  COLLECTION  AND  POST  PROCESSING 

MITRE  Corporation  in  Woodbridge,  VA  coordinated  over  150,000  total 
production  runs  for  this  analysis.  MANA  iterations  run  on  site  produced  the  preliminary 
data.  The  second  data  set  was  completed  during  Project  Albert  International  Workshop  8 
(PAIW  8)  in  Singapore.  The  final  set  of  runs  was  completed  utilizing  the  assets  at  the 
Maui  High  Performance  Computing  Center  (MHPCC).  MITRE  facilitated  the  execution 
of  each  experimental  design  ensuring  proper  implementation  at  the  appropriate  site. 

Each  experiment  returned  a  comma  delimited  file  easily  converted  into  an  Excel 
spreadsheet  or  JMP  statistical  discovery  software  fonnat.  Both  programs,  from  Microsoft 
and  the  SAS  institute  respectively,  are  commonly  used  for  data  manipulation  and 
analysis.  (JMP  User’s  Manual,  2002)  The  output  from  each  run  includes  the  variable 
levels,  duration  of  the  run,  and  the  number  of  killed  agents,  classified  by  type.  The 
number  recorded  in  each  run  is  equivalent  to  the  number  classified,  as  previously 
described  in  Chapter  II.  Spreadsheet  calculations  easily  turn  the  MANA  units  back  into 
real  world  values  for  effective  analysis  in  a  user  friendly  format. 

The  MOE  is  expected  enemy  classification  proportion  per  hour.  This  value  is 
computed  for  each  MANA  run  from  the  output  as  follows.  First,  the  number  of  enemy 
classified  in  that  run  is  divided  by  the  total  number  of  enemy  in  the  scenario.  This  is 
done  to  provide  the  proportion  classified  for  all  130,680  runs.  The  proportion  classified 
is  divided  by  the  actual  run  time  for  each  run  to  yield  the  proportion  classified  per  hour. 
The  last  computation  is  average  classification  proportion  per  hour  across  the  40  runs  at  all 
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3267  design  points.  This  number  provides  the  average  proportion  of  enemy  classified  per 
hour  for  the  given  factor  levels  in  that  design  point — including  controllable  and 
uncontrollable  factors. 

The  statistics  of  the  aggregated  data  over  all  controllable  factors  for  the  one  UAV 
scenario  are  displayed  in  Figure  17.  The  average  proportion  of  enemy  classified  per  hour 
over  all  runs  is  0.0293. 


Figure  17.  Distribution  of  the  MOE,  mean  proportion  of  enemy  classified  per  hour, 
for  the  one  UAV  scenario.  Notice  the  mean  is  2.93%  of  the  enemy  classified  per 

hour. 


Using  40  iterations  at  each  design  point  ensures  that  the  distribution  function  for 
the  random  variable  (mean  proportion  detected  per  hour)  representing  the  MOE  for  a 
particular  data  point  is  approximately  normal,  as  stated  in  the  Central  Limit  Theorem. 
This  ensures  the  data  meets  one  of  the  general  assumptions  for  regression  analysis. 
Randomly  selected  design  points  evaluated  using  the  Shapiro-Wilk  Test  for  Normality 
verify  normality  of  the  response  and  its  use  as  our  estimator  of  the  true  effects  of  the 
particular  parameter  settings.  (Conover,  1999) 

Figure  18  is  a  Normal  Quantile  Plot  of  the  data  for  a  randomly  selected  design 
point.  Data  that  is  normally  distributed  tends  to  fall  on  the  diagonal  line.  Data  contained 
within  the  confidence  interval  bounds  is  said  to  be  distributed  normally  with  95% 
confidence.  (JMP  User’s  Manual,  2000)  Notice  that  this  data  falls  almost  entirely  on  the 


40 


line  and  is  all  contained  within  the  confidence  intervals.  Thus,  at  this  typical  design 
point,  the  MOE  fits  well  to  a  normal  distribution. 


Figure  18.  Normal  Quantile  Plot  of  resulting  enemy  classification  proportion  per  hour 
from  a  randomly  selected  design  point,  number  2678.  Most  of  the  data  fall  on  the 
diagonal  and  all  fall  within  the  95%  confidence  interval  indicating  the  normality  of 
the  measure  of  effectiveness  for  a  typical  design  point. 


After  data  post-processing,  the  resulting  spreadsheet  includes  a  row  for  each 
design  point  listing  the  factor  levels  and  the  associated  mean  enemy  classification 
proportion  per  hour.  The  data  are  now  ready  for  multiple  regression  and  decision  tree 
analysis. 


B.  MULTIPLE  REGRESSION  ANALYSIS 

Multiple  regression  is  a  common  technique  for  determining  the  effect  of  various 
factors  on  a  response  variable.  It  involves  applying  linear  combinations  of  the 
coefficients  of  the  factors  that  predict  the  response  variable  by  minimizing  error. 
Minimizing  the  error  tenn  produces  an  accurate  fit  of  the  response  based  on  the  factors. 
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Various  statistical  packages  are  available  for  facilitating  multiple  regression  analysis. 
JMP  Statistical  Discovery  Software  version  5.0.1a  (JMP  User’s  Manual,  2002)  is  utilized 
for  this  work. 

A  detailed  description  of  the  data  analysis  process  follows  for  the  one  UAV 
scenario  and  is  similar  for  each  model.  The  details  of  the  models  developed  for  the  two 
and  three  UAV  scenarios  can  be  found  in  Appendix  B.  Due  to  anomalies  with  the  multi- 
UAV  runs  requiring  further  iterations  and  time  constraints  on  this  study,  further  analysis 
is  recommended  for  the  multi-UAV  scenarios. 

The  single  UAV  model  utilizes  1089  responses  to  regress  the  controllable  and 
uncontrollable  variables  on  the  MOE,  mean  classification  proportion  per  hour.  This  first 
considers  all  main  effects,  two-way  interaction  tenns,  and  main  effects  quadratic  terms 
resulting  in  a  total  of  76  candidate  terms  for  consideration  in  the  model.  Stepwise 
regression  pairs  down  the  parameter  space  to  only  those  factors  with  a  specified 
significance  level  by  incrementally  adding  and  deleting  terms  to  the  regression  model. 
Once  the  statistically  significant  factors  are  identified,  to  obtain  a  parsimonious  model, 
additional  factors  may  be  removed  if  they  provide  minimal  improvement  to  the  fit.  (JMP 
User’s  Manual,  2000) 

During  the  development  of  the  model,  performing  stepwise  regression  on  the 
factors  eliminates  those  factors  below  the  0.01  level  of  significance.  The  resulting  model 
provides  parameter  estimates  for  the  significant  factors  and  an  R-squared  value  for  the 
entire  model.  The  R-squared  value  is  the  proportion  of  the  variation  in  the  MOE 
explained  by  the  model.  An  R-squared  value  of  1.0  means  that  the  model  perfectly  fits 
the  data.  At  this  point,  an  iterative  process  of  removing  the  tenn  with  the  least 
significance,  reconstructing  the  model,  and  evaluating  the  resulting  fit  of  the  model 
provides  a  means  for  selecting  a  preferred  model.  The  preferred  model  provides  a 
balance  between  simplicity  and  goodness  of  fit. 

Figure  19  demonstrates  this  process  for  the  single  UAV  scenario.  The  R-squared 
value,  or  proportion  of  explained  variation  in  the  MOE,  is  plotted  as  a  function  of  the 
tenns  indicated  along  the  X-axis.  The  line  with  diamonds  represents  the  values  obtained 
from  a  regression  of  controllable  and  uncontrollable  factors,  interactions,  and  squared 
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terms  utilizing  1089  design  points  each  of  which  is  the  mean  of  40  points.  The  line  with 
squares  represents  the  R-squared  values  obtained  from  a  regression  on  just  the 
controllable  factors  using  33  aggregated  design  points. 


R  Squared  by  Terms  for  One  UAV 
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Figure  19.  Graph  of  fit  of  one  UAV  models  by  tenn.  The  figure  shows  the  similarity 

between  the  models  containing  both  controllable  and  uncontrollable  factors  and  the 
models  containing  only  controllable  factors.  Additionally,  the  preferred  model  with 
eight  terms  is  indicated  at  the  point  of  diminishing  return  in  the  fit  on  the  graph.  Note 
that  all  terms  retained  are  controllable  and  there  are  no  interactions  between 
controllable  and  uncontrollable  factors.  [Best  viewed  in  color] 


Notice  the  similarity  between  the  two  plots  and  how  closely  they  follow  each 
other.  This  is  a  visual  representation  of  the  lack  of  practical  significance  of  the 
uncontrollable  variables  in  this  scenario  over  the  ranges  explored.  Analysis  of  each  of 
the  other  two  models,  for  two  and  three  UAVs  produces  similar  results.  The  preferred 
model,  even  with  the  uncontrollable  factors  considered,  still  contains  only  controllable 
factors.  This  leads  to  a  simpler  model  focusing  on  just  the  controllable  variables  and 
allows  for  the  aggregation  of  the  data  from  1089  points  to  33  points  without  significant 
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loss  of  information.  The  remaining  analysis  considers  the  mean  enemy  classification 
proportion  per  hour  for  the  40  times  33  =  1320  runs  at  each  controllable  factor  design 
point. 


Figure  19  also  displays  the  distinct  point  of  diminishing  returns  as  factors  are 
added  to  the  model.  This  typical  characteristic  of  regression  models  is  useful  in 
determining  the  difference  between  statistical  significance  and  practical  significance. 
While  all  the  terms  listed  at  the  bottom  of  the  graph  are  statistically  significance  at  the 
0.01  level,  they  provide  minimal  additional  explanation  of  the  response  after  the  first 
seven  or  eight  terms.  The  first  eight  terms  listed  along  the  bottom  of  the  graph  explain 
over  90%  of  the  variability  in  mean  enemy  classification  proportion  per  hour  across  all 
the  levels  explored  in  this  scenario. 


Another  way  of  looking  at  the  effects  of  each  tenn  is  to  add  them  into  the  model 
in  reverse  order.  Figure  20  shows  the  incremental  predictive  power  each  term  on  the 
horizontal  X-axis  contributes  to  the  model  in  terms  of  R-squared  in  the  response  variable 
on  the  vertical  Y-axis.  The  model  from  all  the  factors  is  the  line  with  diamonds  and  the 
model  obtained  from  the  aggregated  data  for  the  controllable  factors  is  the  line  with 


squares. 


Figure  20.  Proportion  of  variation  in  one  UAV  mean  enemy  classification  proportion 

per  hour  across  all  scenario  factor  levels  as  each  term  is  added  to  the  model.  There  is 
a  clear  point  of  diminishing  return  and  similarity  between  the  models  with  all  factors 
and  the  models  aggregated  over  the  uncontrollable  factors.  [Best  viewed  in  color] 
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Again,  the  resemblance  between  the  two  plots  clearly  depicts  the  similarity 
between  the  two  models.  Using  the  simpler  model  developed  from  controllable  factors 
and  aggregated  data  produces  no  significant  loss  of  information.  In  addition,  this  graph 
more  clearly  depicts  the  point  of  diminishing  return.  The  value  of  adding  the  next  most 
significant  term  after  the  eighth  tenn  is  minimal.  The  arrow  indicates  the  desired  point  of 
balance  between  goodness  of  fit  and  simplicity.  The  preferred  eight  term  model  contains 
the  following  terms  for  the  single  UAV  scenario  in  order  of  importance: 

1 .  UAV  Sweep  Width 

2.  UAV  Probability  of  Classification 

3.  UAV  Sweep  Width  Squared 

4.  Maximum  Time  Available 

5.  Reactivity 

6.  Interaction  between  UAV  Sweep  Width  and  Reactivity 

7.  Maximum  Time  Available  Squared 

8.  UAV  Speed 

Note  that  all  these  tenns  are  controllable  factors. 

UAV  sweep  width  is  the  most  significant  predictor  of  the  variability  in  the  MOE 
across  UAV  capabilities  in  this  scenario.  Nearly  40%  of  the  variance  in  the  proportion  of 
enemy  classified  per  hour  is  explained  by  the  UAV’s  sweep  width  alone.  The  first  four 
tenns  explain  over  80%  of  the  variance  in  the  MOE.  UAV  sweep  width,  probability  of 
classification,  sweep  width  squared,  and  maximum  time  available  provide  a  good 
model — explaining  the  vast  majority  of  the  variability  all  by  themselves. 

There  is  a  noticeable  dip  above  the  reactivity  tenn  in  Figure  20,  due  to  the  fact 
that,  on  its  own,  reactivity  does  not  explain  a  significant  portion  of  the  variability. 
However,  the  next  term,  an  interaction  between  reactivity  and  the  UAV’s  sweep  width,  is 
a  significant  predictor  of  the  variability  in  the  MOE.  It  is  traditional  to  include  the  main 
effect  tenn  in  any  model  where  an  interaction  term  is  significant  even  if  the  main  effect 
tenn  is  not.  (Whittaker,  2003)  Thus,  reactivity  is  retained  in  this  model  even  though  it  is 
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not  mathematically  required.  The  squared  effects  of  maximum  time  available  and  UAV 
speed  also  have  practical  significance,  although  not  as  great  as  the  previous  terms. 

Figure  21  is  a  visual  representation  of  the  preferred  model.  This  plot  of  the 
predicted  versus  actual  response  displays  how  closely  the  model  explains  the  MOE. 
Notice  how  well  the  data  points  follow  the  diagonal — indicating  a  nicely  fit  model.  The 
residual  plot  on  the  right  displays  the  homoscedasticity,  or  constant  error  variance,  lack 
of  influential  cases,  normal  residual  distribution,  and  linear  relationship.  (Hamilton, 
1992)  The  associated  statistics  for  this  model  are  provided  in  Figure  22.  The  actual 
coefficients  for  each  tenn  in  the  model  are  listed.  Also  of  note  is  the  way  in  which  the 
interaction  terms  and  square  terms  are  evaluated.  The  mean  for  each  is  subtracted  off  for 
each  factor  included  in  an  interaction  or  squared  term  to  ensure  proper  scaling. 
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Figure  2 1 .  Predicted  versus  actual  mean  enemy  classification  proportion  per  hour 
displaying  the  good  fit  of  the  preferred  single  UAV  model  with  eight  terms  and 
associated  residual  plot  verifying  the  absence  of  pattern  in  the  residuals. 
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Summary  of  Fit 

RSquare 
RSquare  Adj 
Root  Mean  Square  Error 
Mean  of  Response 
Observations  (or  Sum  Wgts) 

Analysis  of  Variance 

Source  DF  Sum  of  Squares 

Model  8  0.00486708 

Error  24  0.00037337 

C.  Total  32  0.00524045 

Parametei  Estimates 


0.928752 

0.905003 

0.003944 

0.029291 

33 


Mean 

0. 

0. 


Square  F  Ratio 
000608  39.1067 

00001 6  Prob  >  F 
<.0001 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

0.0047896 

0.003806 

1.26 

0.2203 

UAVRSpeed 

0.0000198 

0.000007 

2.69 

0.0128 

UAVRSW 

0.0000031 

2.764e-7 

11.27 

<0001 

UAVPCIass 

0.0243387 

0.002762 

8.81 

<.0001 

UAVUnkAttr 

-0.000008 

0.000028 

-0.27 

0.7876 

MaxSteps 

-0.001322 

0.000221 

-5.98 

<0001 

(UA  VRSW-6058 .1 2)*(UAVUnkAttr-50) 

-3.652e-8 

1 .1 38e-8 

-3.21 

0.0038 

(UA  VRSW-6058 .1 2)*(U  A  VRSW-6058 .1 2) 

-7.03e-10 

1.36e-10 

-5.15 

<.0001 

(MaxSteps-7  00242)*(MaxSteps-7 .00242) 

0.0002559 

0.000084 

3.05 

0.0056 

Figure  22.  Preferred  one  UAV  model.  The  R-squared  value  for  this  model  is  above 
90%.  The  display  shows  the  coefficients  for  each  term  and  the  significance  of  each  of 

the  terms  as  well  as  the  overall  model. 


This  model  makes  sense.  As  UAV  speed,  sweep  width,  and  sensor  capability 
increase,  the  proportion  of  enemy  classified  per  hour  increases.  These  effects  are 
intuitive,  as  capabilities  increase,  so  does  performance,  lending  credibility  to  the  model. 
Conversely,  as  the  time  the  UAV  spends  in  the  area  increases,  the  proportion  of  enemy 
classified  per  hour  decreases,  requiring  slightly  more  contemplation  but  also  making 
sense.  The  longer  the  UAV  searches,  new  contacts  become  sparse.  Additionally,  the 
target  rich  environments  are  visited  first,  providing  more  opportunity  for  classifications. 
This  also  makes  tactical  sense  since  it  is  quite  reasonable  to  send  a  search  asset  to  survey 
the  most  likely  enemy  sites  first.  The  rate  at  which  classifications  occur  decreases  over 
time. 

The  interaction  term  indicates  the  significant  relationship  between  sweep  width 
and  reactivity.  It  may  seem  better  to  have  a  wide  sweep  width.  However,  when  the 
sweep  width  is  high  and  the  UAV  is  employed  with  a  propensity  to  follow  unidentified 
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contacts  until  classified,  the  greater  sweep  width  can  accumulate  too  many  unknown 
contacts.  This  has  a  negative  effect  on  the  proportion  classified  per  hour  due  to  a  conflict 
of  interests.  The  UAV  cannot  follow  all  unidentified  contacts  in  this  case  and  performs 
poorly. 

Figure  23  displays  this  interaction  in  the  top-middle  and  left-center  row  plots.  For 
two  different  levels  of  reactivity,  10  and  90  for  example,  varying  the  sweep  width  has  a 
different  effect.  This  is  depicted  in  the  left  center  plot  by  the  non-parallel  curves.  The 
red  curved  line,  with  a  value  of  10,  indicates  minimal  desire  to  chase  unclassified 
contacts  and  a  desire  to  stay  on  the  route.  In  this  case,  as  sweep  width  is  increased,  the 
point  of  diminishing  return  is  reached  later  than  in  the  case  depicted  by  the  blue  line.  The 
blue  line,  with  a  value  of  90,  indicates  a  high  propensity  to  follow  unclassified  contacts 
vice  follow  routing.  When  the  UAV  is  more  inclined  to  follow  the  unclassified  contacts, 
a  point  of  diminishing  return  is  reached  more  quickly  because  it  develops  too  many 
competing  interests  more  quickly. 


Figure  23.  Interaction  plots  between  UAV  sweep  width,  UAV  probability  of 
classification,  and  max  steps.  The  display  shows  the  presence  of  an  interaction 
between  SW  and  PClass  and  the  nonlinear  effects  of  SW  and  time.  [Best  viewed  in 

color] 
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In  the  top  middle  plot,  the  effect  of  varying  reactivity  is  different  for  a  sweep 
width  of  2,067  meters,  in  red,  compared  to  a  sweep  width  of  10,040  meters,  in  blue.  The 
difference  in  slopes  between  the  two  lines  is  clear.  Increasing  the  reactivity  of  the  UAV 
with  the  large  sweep  width  has  a  negative  effect.  The  wider  the  sweep  width  becomes, 
the  more  unclassified  contacts  the  UAV  detects,  and  the  harder  it  is  to  classify  each.  The 
UAV  is  overtasked.  Conversely,  when  the  sweep  width  is  low,  increasing  the  reactivity 
helps  the  under  tasked  UAV  find  more  enemy  contacts. 

The  sweep-width-squared  term  indicates  a  point  of  diminishing  returns  for  sweep 
width.  As  sweep  width  increases,  holding  all  other  factors  constant,  the  mean  enemy 
proportion  classified  per  hour  increases  up  to  a  certain  point.  This  makes  intuitive  sense. 
As  discussed  in  the  previous  chapter,  increasing  the  sweep  width  without  increasing  the 
capability  of  the  sensor  to  achieve  the  equivalent  single  glimpse  probability  of 
classification  may  result  in  a  decrease  in  perfonnance  at  some  point. 

The  squared  tenn  for  time  the  UAV  is  on  station,  indicates  that  there  is  not  just  a 
simple  linear  relationship  explained  by  the  main  effect  term,  but  that  endurance  has  a 
point  of  diminishing  return  as  well.  Figure  23  displays  the  interactions  of  endurance  as 
dashed  lines  indicating  no  significant  interaction  with  the  other  variables.  Each  plot  runs 
parallel  with  respect  to  the  interaction  term.  However,  the  nonlinear  effect  of  “max 
steps”  can  clearly  be  seen.  In  both  plots  on  the  right  side  of  Figure  23,  as  max  steps 
varies,  the  mean  classification  proportion  per  hour  decreases  in  a  curvilinear  fashion. 
Although  the  mean  proportion  of  enemy  classified  per  hour  decreases  over  time,  the  rate 
of  decrease  diminishes  as  well.  It  appears  that  about  seven  hours  is  the  point  at  which  the 
curve  levels  off. 

The  significance  of  each  of  these  factors  can  be  visualized  using  leverage  plots. 
The  leverage  plot  shows  for  each  point  what  the  residual  would  be  with  and  without  that 
effect  in  the  model.  (JMP  User’s  Manual,  2002)  Figure  24  displays  the  effect  each  term 
has  on  mean  proportion  of  enemy  classified  per  hour.  As  the  factor  level  is  varied  across 
the  bottom,  while  holding  all  other  terms  constant,  the  effect  on  the  MOE  can  be 
referenced  on  the  left.  The  terms  with  a  greater  effect  have  a  line  with  a  steeper  slope. 
Sweep  width  has  the  largest  impact  and  appears  with  the  greatest  slope.  Probability  of 
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classification,  the  second  most  significant  factor,  is  very  close  in  significance,  as  the 
associated  plot  in  Figure  24  is  nearly  as  steep  as  the  sweep  width  plot.  In  this  manner, 
relative  significance  can  be  visualized  for  each  of  the  effects. 


Leverage  Plots  for  One  UAV  8  Term  Model 


*  UAVRSW'UAVRSW  _ |  ▼  *  M.ixSteps'M.ixSteps 


Figure  24.  Leverage  plots  of  one  UAV  preferred  model  terms  indicating  degree  to 
which  each  tenn  affects  the  MOE,  mean  proportion  of  enemy  classified  per  hour. 

C.  CLASSIFICATION  AND  REGRESSION  TREES 

Classification  and  Regression  Trees  (CRTs)  are  good  tools  for  creating  decision 
trees  and  provide  another  way  to  analyze  the  relationship  between  factors  and  the  MOE. 
A  regression  tree  is  a  recursive  partition  of  the  raw  data  into  sets  of  inputs  containing 
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similar  responses.  Partitioning  of  the  data  occurs  successively  according  to  the  optimal 
splitting  value  determined  from  all  possible  values  of  each  available  variable.  The 
optimal  splitting  value  is  the  value  of  the  predictor  variable  that  minimizes  sum  of  square 
error  amongst  all  predictors.  After  each  split,  the  next  optimal  split  is  determined  within 
each  partition.  This  may  be  the  same  variable  as  the  initial  split  or  a  different  variable 
obtained  from  all  available  factors  and  can  be  different  for  each  partition.  Considering 
each  partition  independently  of  the  previous  splits  automatically  accounts  for 
interactions.  This  continues  until  the  improvement  in  fit  falls  below  user  specified  levels. 
Again,  we  must  balance  fit  with  parsimony.  The  concept  is  complex,  but  the  resulting 
model  is  easy  to  understand. 

Figure  25  displays  a  recursive  split  of  the  raw  data  from  all  43,560  MANA  runs 
on  all  controllable  factors  and  uncontrollable  factors  for  the  one  UAV  scenario.  As 
partitioning  of  the  data  proceeds,  the  most  significant  factors  produce  the  first  categories. 
The  splitting  point  for  a  factor  range  suggests  an  upper  or  lower  limit  for  that  factor 
capability  producing  significant  improvement  in  the  MOE.  Each  box  indicates  the 
optimal  factor  and  the  optimal  level  to  divide  upon.  Details  within  the  box  include  the 
number  of  data  points  within  the  split,  the  mean  enemy  classification  proportion  per  hour 
and  the  standard  deviation  within  the  split. 

This  analysis  complements  the  regression  analysis  in  the  previous  section.  Again, 
all  of  the  terms  in  the  tree  are  controllable  factors.  The  first  split  is  made  on  UAV  sweep 
width.  This  is  the  single  most  significant  factor  in  this  scenario.  The  decision  tree  also 
provides  the  optimal  split  point  at  about  4,430  meters.  Based  on  the  analysis  for  this  Sea 
Viking  scenario,  over  the  ranges  examined,  a  system  with  a  sweep  width  greater  than 
4,430  meters  can  be  expected  to  provide  a  rate  of  enemy  proportion  classified  over  two 
times  greater,  on  average,  than  a  system  with  sweep  width  below  4,430  meters,  0.0348 
and  0.0166  respectively. 
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Figure  25.  Decision  tree  split  on  the  raw  data  by  proportion  of  enemy  classified  per 
hour  for  each  MANA  run  of  the  one  UAV  scenario,  considering  controllable  and 
uncontrollable  factors.  The  tree  indicates  the  overall  significance  of  sweep  width  and 
probability  of  classification,  and  the  interaction  with  reactivity. 


The  next  most  important  factor  is  probability  of  classification.  If  sweep  width  is 
above  the  optimal  minimum  of  4,430  meters,  ensuring  the  sensor  capability  provides  at 
least  a  40%  probability  of  classification  for  the  expected  environmental  and  geographic 
conditions  will  provide  over  a  60%  increase  in  the  expected  proportion  of  enemy 
classified  per  hour. 

The  next  significant  factor  explaining  performance  is  reactivity  when  the  UAV 
sweep  width  is  less  than  4,430  meters.  In  this  case,  employing  a  more  reactive  UAV  will 
increase  the  proportion  of  enemy  classified  per  hour.  The  additional  information  gained 
from  following  unclassified  contacts  may  be  acted  upon  and  provides  an  increase  in 
performance. 

Two  factors  are  noticeably  missing  from  the  CRT:  UAV  Speed  and  Time.  These 
factors  appeared  in  the  regression  analysis,  but  even  when  this  CRT  extends  several  more 
levels,  these  factors  do  not  show  up  as  might  be  expected.  Their  significance  is 
substantially  less  than  the  primary  factors  brought  out  by  the  decision  tree. 
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This  brings  up  the  possibility  of  another  MOE,  namely  proportion  of  enemy 
classified  per  mission.  This  MOE  captures  the  effect  of  the  entire  mission  whereas  the 
original  MOE  captures  the  time  effect  as  a  rate.  By  considering  the  total  enemy  classified 
during  a  mission,  the  effect  of  time  may  become  more  apparent. 

With  this  new  MOE,  or  MOE  2,  proportion  of  enemy  classified  per  mission,  a 
new  CRT  tree  is  developed  and  appears  in  Figure  26  below.  As  expected,  time,  in  this 
case  max  time,  is  the  primary  factor  when  considering  the  total  amount  of  enemy 
classified  during  a  mission.  A  UAV  on  station  at  least  seven  hours  will  classify  nearly 
twice  the  proportion  of  the  enemy  than  a  UAV  with  fewer  than  seven  hours  on  station 
time  when  averaged  over  all  the  other  variables.  Additionally,  given  seven  hours  to 
search,  much  greater  probability  of  classification  is  required  to  produce  a  significant 
increase  in  classifications.  This  suggests  endurance  may  make  up  for  short  comings  in 
sensor  capability. 


Figure  26.  CRT  split  on  the  raw  data  by  proportion  of  enemy  classified  per  mission 
for  each  MANA  run  of  the  one  UAV  scenario,  considering  controllable  and 
uncontrollable  factors.  The  tree  indicates  the  significant  time  effect  and  the 
appearance  of  speed  while  retaining  previous  decision  factors. 


53 


The  effect  of  sensor  capability,  namely  sweep  width  and  probability  of 
classification,  appear  similarly  in  this  model  as  they  did  in  the  previous  CRT.  A  brief 
study  of  the  second  decision  tree  in  Figure  26  reveals  that,  on  average,  a  UAV  with  a 
sweep  width  greater  than  4,200  meters  and  a  probability  of  classification  of  0.4  or  more 
can  classify  approximately  18%  of  the  enemy  in  this  scenario  in  about  four  hours.  This 
provides  a  good  reference  point  in  determining  factor  effects. 

Speed  appears  in  Figure  26  when  the  UAV  is  going  for  over  seven  hours  and  has 
a  probability  of  classification  less  than  0.7.  Compared  to  the  other  splits,  the  effect  is 
minimal  making  it  the  sixth,  and  last,  one  to  appear.  At  speeds  over  160  knots  the 
proportion  of  enemy  classified  increases  by  less  than  30%  on  average.  A  sweep  width 
greater  than  about  4,200  meters  with  fewer  than  seven  hours  available  nearly  triples  the 
expected  proportion  of  enemy  classified. 
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CONCLUSIONS 


The  purpose  of  computing  is  insight,  not  numbers. 

-Richard  Hamming 


A.  ANALYSIS  SUMMARY 

The  data  obtained  from  the  design  of  experiments  and  the  Sea  Viking  scenario 
implemented  in  MANA  for  this  study  provide  normally  distributed  data  points  from 
which  to  conduct  statistical  analysis.  The  uncontrollable  factors,  or  noise,  imposed  in  this 
scenario  produce  statistically  significant  effects  on  the  MOE,  but  not  practical  effects. 
This  allows  for  simplification  of  the  data  into  aggregated  points  over  the  noise  and  further 
analysis  which  focuses  on  the  controllable  variables. 

This  study  uses  two  analysis  techniques  to  look  at  the  proportion  of  enemy 
classified  per  hour:  multiple  regression  analysis  and  Classification  and  Regression  trees. 
The  two  analyses  complement  each  other.  Each  analysis  identifies  similar  factors  of 
greatest  importance,  key  interactions,  and  provides  similar  insights.  In  contrast,  the 
regression  analysis  yields  formulae  for  predicting  UAV  performance  for  capability 
combinations  not  explicitly  modeled  in  the  simulation.  Additionally,  relative  effects  of 
one  capability  set  can  be  compared  to  others  using  the  infonnation  from  regression 
analysis.  The  Classification  and  Regression  Tree  analysis  provides  a  hierarchical  view  of 
the  factors.  The  splits  define  factor  levels  as  a  minimum  or  maximum  goal  to  keep  in 
mind. 

The  regression  analysis  on  this  aggregate  data  produces  a  good  fitting  model  for 
the  single  UAV  scenario.  Sweep  width  and  probability  of  classification  have 
dramatically  more  significant  effects  on  the  proportion  of  enemy  classified  per  hour  than 
the  other  factors  for  the  ranges  and  factors  considered.  This  makes  sense  and  lends 
credibility  to  the  modeling.  Interactions  between  the  factors  indicate  the  degree  to  which 
the  factors  must  be  considered  together.  As  expected,  sweep  width,  probability  of 
classification  or  sensor  capability,  and  reactivity  or  employment  philosophy  are  related 
and  insight  is  provided  regarding  these  relations. 
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Decision  points  for  primary  factors  are  indicated.  A  sweep  width  above  about 
4,500  meters  provides  a  significant  increase  in  the  proportion  of  enemy  classified  per 
hour.  A  sensor  package  providing  at  least  0.4  probability  of  classification  per  glimpse  for 
contacts  within  its  intended  environment  will  significantly  improve  perfonnance  with 
regard  to  the  proportion  of  enemy  classified  per  hour. 

The  absence  of  time  and  speed  in  the  CRT  led  to  the  value  of  a  second 
performance  measure:  proportion  of  enemy  classified  per  mission.  The  decision  tree 
based  on  this  MOE  brings  to  light  the  value  of  having  a  UAV  with  an  on  station  time  of 
at  least  seven  hours.  The  significance  of  speed  is  once  again  revealed  as  minimal  in  this 
scenario. 

Using  the  analytical  model  obtained  from  this  analysis,  predictions  of  the  relative 
performance  expected  for  other  capability  sets  in  this  scenario  can  be  evaluated.  The 
decision  points  provide  a  threshold  value  to  keep  in  mind  in  the  development  of  systems 
with  regard  to  relative  expected  performance  in  this  scenario.  By  decomposing  the  data 
according  to  the  decision  points,  further  analysis  can  provide  a  more  accurate  picture  of 
the  effects  of  the  factors  on  the  MOE. 

Minor  anomalies  are  present  in  the  two  and  three  UAV  scenario  which  warrant 
additional  analysis  and  iterations.  Models  based  on  the  current  infonnation  are  in 
Appendix  B.  They  are  consistent  with  the  insights  for  the  single  UAV  scenario. 


B.  KEY  TACTICAL  INSIGHTS 

The  most  important  factor  when  considering  time  sensitive  Intelligence 
Preparation  of  the  Battlespace  in  the  Sea  Viking  Scenario  is  the  sweep  width  of  the  UAV. 
In  general,  wider  sweep  widths  yield  higher  expected  proportions  of  enemy  classified 
each  hour — often  more  than  twice  as  much.  This  is  qualified  by  the  assumption  that  the 
sensor  package  can  maintain  a  fairly  high  probability  of  classification  as  the  sweep  width 
increases.  In  less  than  seven  hours,  a  UAV/sensor  package  capable  of  producing  a 
probability  of  classification  of  at  least  0.4  over  a  4,500  meter  sweep  width  may  be 
expected  to  produce  rate  of  enemy  classification  nearly  three  times  greater,  on  average, 


56 


than  a  UAV  that  does  not  meet  these  standards  for  the  scenario  detailed  in  this  study. 
This  may  be  crucial  in  a  time  constrained  situation. 

In  consideration  of  mission  success,  or  classifying  the  largest  proportion  of  the 
enemy  during  the  course  of  a  mission,  a  UAV  on  station  at  least  seven  hours  is  most 
valuable  in  this  scenario.  This  time  on  station  may  mitigate  many  of  the  shortcomings  in 
sensor  capability  or  speed  of  the  UAV.  On  the  other  hand,  with  fewer  than  seven  hours 
time  on  station,  the  factors  discussed  above  for  time  sensitive  gathering  of  intelligence 
provide  the  greatest  proportion  of  enemy  classified  per  mission  in  the  Sea  Viking 
scenario. 

Whether  considering  a  rate  of  classification  or  the  proportion  classified  for  an 
entire  mission,  use  of  tactical  routing  is  more  effective  than  traditional  search  patterns. 
This  makes  tactical  sense  and  lends  credibility  to  the  model.  For  employment 
considerations,  the  more  intelligence  that  is  available  the  more  important  it  is  to  follow 
routing  as  opposed  to  chasing  unclassified  contacts.  The  balance  between  reactivity  and 
strictly  following  a  route  is  difficult  to  quantify.  With  that  caveat,  reserving  about  one- 
third  of  the  on  station  time  for  chasing  unknowns  and  using  the  remainder  to  follow  a 
tactical  route  appears  to  be  the  best  combination  for  the  Sea  Viking  scenario.  A  large 
sweep  width  and  low  probability  of  classification  may  result  in  too  much  wasted  time  if 
reactivity  is  high.  Conversely,  high  reactivity  can  be  effective  if  the  sweep  width  is  low. 

C.  ADDITIONAL  INSIGHTS 

Many  insights  have  been  realized  in  the  course  of  this  analysis.  The  most 
significant  appear  in  the  previous  section.  The  following  is  a  list  of  additional  insights 
surfacing  during  this  work. 

•  Speed  has  a  positive  effect  on  UAV  classification  perfonnance.  However, 
speeds  greater  than  200  knots  provide  little  improvement  in  ability  to 
classify  enemy.  Certainly  there  are  other  important  reasons  to  have  a  fast 
UAV  which  should  be  considered,  for  example  dash  speed. 

•  Increasing  sweep  width  when  minimal  enemy  intelligence  is  available  will 
increase  the  proportion  of  enemy  classified.  However,  there  is  a  point  of 
diminishing  return  when  task  saturation  becomes  an  issue.  This  is 
especially  true  if  the  probability  of  classification  is  too  low. 
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•  Two  UAVs  do  not  provide  twice  the  classification  ability;  however  there 
is  improvement  over  a  single  UAV.  Three  UAVs  seems  to  have  a  more 
synergistic  effect,  doubling  the  expected  proportion  classified  with  two 
UAVs  in  this  scenario  given  the  routes  examined. 

•  Agent-based  models  and  data  farming  techniques  provide  an  efficient 
means  to  view  the  effects  of  a  variety  of  parameters.  Unknown  values 
may  be  farmed  to  provide  insights  to  parameter  effects  without  explicit 
modeling  of  capabilities  which  may  be  unknown  and  previously  guessed. 

•  Creative  modeling  is  required  in  Agent-based  Modeling.  The  point  is 
quick  turnaround  and  insights  into  interactions  and  focusing  further 
analysis.  Using  readily  available  MOEs  in  a  particular  ABM  can  enable 
more  effective  and  capable  analysis. 


D.  FOLLOW  ON  WORK 

The  following  is  a  list  of  follow  on  research  of  value  that  could  be  accomplished 
utilizing  this  work: 

•  Analysis  of  factors  effecting  classification  of  time  critical  targets 

•  Analysis  of  effects  of  the  terrain 

•  Data  analysis  on  Multi-UAV  data 

•  Further  development  and  enhancement  of  the  Excel  tool  for  parameter 
exploration 

•  Focused  analysis  over  the  key  parameters  and  ranges  identified 

•  Analysis  of  the  effect  of  a  much  larger  neutral  to  enemy  ratio 

•  Repeat  analysis  in  an  OIF  based  scenario 


The  following  is  a  list  of  follow  on  research  of  value  stemming  from  this  research: 

•  Validation/comparison  of  the  MANA  Sea  Viking  scenario  with  a  similarly 
developed  scenario  in  Combat  XXI 

•  Development  of  a  model  addressing  the  dynamic  retasking  issue 

•  Development  of  a  model  addressing  the  multiple  concurrent  operations 
issue 

•  Development  of  a  model  focusing  on  the  Global  Hawk  Maritime 
Demonstration  for  the  Maritime  Patrol  and  Reconnaissance  Force 

•  Analysis  of  the  available  sensor  packages  and  development  of  expected 
probabilities  of  classification 
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•  Reliability  study  determining  the  number  of  UAVs  required  to  support  a 
Sea  Viking  scenario 

•  Human  factors  study  detennining  effective  screening  and  classification 
techniques  integrating  the  man  and  the  machine 

•  Analysis  of  distributed  communications  flow  for  a  network  centric 
environment  incorporating  multiple  UAVs  for  sensing  and  shooting 
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APPENDIX  A.  SEA  VIKING  04  SCENARIO  DETAILS 


The  following  slides  are  taken  directly  from  the  Sea  Viking  Scenario  brief  and 
describe  the  enemy  situation  for  the  scenario.  (Marine  Corps  Warfighting  Lab, 
2003) 


Game/Exercise  Assumptions 

•  Play  will  remain  at  the  JTF  level  and  below, 
regional  powers  are  not  and  will  not  become 
hostile 

•  Limited  Host  Nation  Support,  no  port  or 
airfield  in  JOA 

•  Very  restricted  basing  in  theater 

•  Timeframe:  April  or  October?  2015 

•  Area  of  Play:  SE  Asia 

•  Classification:  Unclassified 

•  Force  Levels:  MEB  /  ESF  equivalent 

•  Game/Exercise  play:  Operational/Tactical 

•  Sea  base  minimum  OTH  (25nm  at  sea) 


Red  Objectives 

•  Goals:  Gain  independence 

•  Methodology: 

-  Survive  until  Central  Government  collapses  “Fait  Accompli” 

-  Avoid  direct  conflict  with  Coalition  forces 

-  Invite  NGOs  into  conflict  area 

-  Invite  international  media,  showcase  civilian  deaths 

-  Repeatedly  stress  “Caliphate”  does  not  desire  hostilities 

-  Ask  for  UN  brokered  cease  fire 

-  Promise  to  act  decisively  to  end  piracy  within  territorial 
waters 

-  Offer  to  hold  elections  in  2  years 

-  Portray  self  as  small  Muslim  nation  attacked  unjustly  by  rich 
Christian  crusaders 

-  If  conflict  unavoidable,  attempt  to  draw  coalition  forces  into  a 
MOUT/Jungle  fight 
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Red  Order  of  Battle 

Air  -  IADS  (mobile  SA-10/20,13,8,  &  6?).  Adding  Ships  with  increased  AD  cap  (HQ-61). 
AAA  -  RBS-90  Bolide,  2S6M  (SP  30mm/SA-19) 

TBM  -  CSS-2  IRBM,  6  mobile  launchers,  2  x  reloads,  6  decoys. 

Air  Force:  72  -  SU-27  Flanker,  and  72  -  SU-25  Frogfoot,  36  -  Foxhounds,  2  B-727  AEW, 
3  KC-130  Refuelers 

Land  -  6  SPF  Brigades.  Mobile  Artillery  (VTT-322,  2S23)  with  smart  sub-munitions  and 
passive  counter  battery  capability  (SORAS).  PT-76  upgraded  thermals,  shoot  on  move, 
90mm.  Decentralized  Infantry  Operations  (Fire  and  Forget  Soldiers). 

Local  Constabulary  and  irregulars.(need  to  establish  #  &  Loc) 

Naval  -  6  Kilo  636  (w  ASCM),  patrol  craft/missile  patrol  craft,  mines,  and  armed  militia 
in  small  civilian  craft. 

Naval  Air  -  8  Folker  27’s  Maritime  Patrol  Aircraft  (Joint  real-time  targeting) 

Naval  Coastal  Defense  -  3  Batteries  mobile  ASCM’s  (YJ-82  ASCM)  1  Naval  Inf  Bde  to 
protect  ASCM  and  base. 

Electro-Magnetic  -  Jamming  /  spoofing  capability  (radio  and  GPS)  Computer  Network 
Attack? 

Space  -  Access  to  commercial  space  assets.  Controls  major  Green  satellite  terminal 

WME  -  Red  commercial  and  research  facilities  are  capable  of  make  biological  and 
chemical  weapons. 


The  following  force  structure  is  for  the  enemy  units  portrayed  in  the  Sea  Viking 
04  scenario.  (Marine  Corps  Warfighting  Lab,  2003) 


Light  Brigade  x  3 


J 6x82mm  A  A 

-  A  A 

|  12  x  AGS  - 17  ^ 


J,  1 8  X  SA-1 8  or  RBS  -  90 


8-ZU-23 
or  DShK 


0% 


9x10Qnmor  84mm Reccilless Rifle 
(vehicle  moulted) 
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Reinforced  Brigade  x  3 


3  Cos  of  3  Btrys  of 

6  x  82mm  7  x  PT76  Lt  Tanks  6x120mmSP 


£  12  x  AGS -17 


ji  18  x  SA-18  or  RBS  -  90 


9  x  106mm  or  84  mm  Recoilless  Rifle 
(vehicle  mounted) 


6 

AAA  Pit 

A  A 
AA 
A  A 
A  A 

8-2S6M 


o  <§> 

Transport  Co 
54  Med  Trucks 
(1  Inf  Bn) 


Naval  Infantry  Brigade  x  1 


BDE  mobilizes  into  4  Bn  TF's 
3  to  defend  ASCM  Batteries 
1  in  reserve/defend  Naval  Base 


Bn  T/E 


Per  Inf  Bn 

425-600  men  Mortar  Bty 


£  18  x  60mm 
J  6  x  82mm 


|  12  X  AGS  - 17 


i 

(A) 


18  x  SA-18  or  RBS  -  90 

9  x  106mm or  84mm  Recdlless  Rile 
(vehidemomted) 


T 


AAA  Pit 


1  k 
k  k 
k  k 
A  k 
k  A 

10-ZU-23 
or  DShK 
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APPENDIX  B.  REGRESSION  MODELS  FOR  MULTI-UAV 

SCENARIOS 


Anomalies  discovered  during  data  analysis  prompted  additional  iterations  of 
various  design  points  for  the  multi-UAV  scenarios.  Due  to  time  constraints  and  the  need 
for  still  more  runs,  detailed  analysis  of  the  multi-UAV  scenarios  is  not  presented  in  this 
work.  Some  insights  from  the  data  may  be  gained,  but  the  statistical  rigor  to  support 
them  is  lacking  at  this  point.  It  is  highly  encouraged  that  further  research  investigates  this 
data. 


The  model  produced  for  the  two  UAV  scenario,  displayed  below,  is  very  similar 
to  the  model  for  the  one  UAV  scenario,  with  a  couple  exceptions.  The  reactivity  terms 
drop  out  earlier  and  are  not  included  in  the  preferred  model  and  a  strong  interaction 
between  sweep  width  and  probability  of  classification  appears.  The  reactivity  terms  are 
statistically  significant,  and  they  could  be  included  in  the  model,  however  they  do  not 
provide  the  level  of  practical  significance  seen  in  the  single  UAV  case.  The  interaction 
between  sweep  width  and  probability  of  classification  appears  similarly  in  the  decision 
tree  analysis  for  the  single  UAV  model. 


Preferred  Model  for  Two  UAV — 6  Terms 

_  0.1 2-i - 7 — 

CU  /•  , 

§0.10 - 

4  0.08-  As  . 

Q. 

g 

%  0  06_ 

ro  •  •  *. ' 

O  0.04— — 

UJ_ 

|  0.02- 
O.OO-f^ 

.00  .02  .04  .06  .08  .10  .12 

Mean(ECIassProp/Hr)  Predicted  P<.0001 
RSq=0.92  RMSE=0.009 


7^7 


Summary  of  Fit 

RSquare 
RSquare  Adj 
Root  Mean  Square  Error 
Mean  of  Response 
Observations  (or  Sum  Wgts) 
Analysis  of  Variance 


0.917467 

0.89842 

0.008964 

0.037937 
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Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 

Model  6  0.02322619  0.003871  48.1706 

Error  26  0.00208938  0.000080  Prob»F 

C.  Total  32  0.02531558  <.0001 


Paiametei  Estimates 

Term 

Intercept 

UAVRSpeed 

UAVRSW 

UAVPCIass 

MaxSteps 

(UAVRSW-6058 .1 2)«(UAVPCIass-0 .5) 
(MaxSteps-7 ,00242)*(MaxSteps-7 .00242) 


Estimate  Std  Error  t  Ratio  Prob>|t| 
-0.065488  0.00779  -8.41  <.0001 

0.0001084  0.000017 
0.0000069  6.281  e-7 
0.0530479  0.006277 
0.0020823  0.000502 
0.0000167  0.000003 
-0.000643  0.000195 


Residual  Plot  of  Preferred  Two  UAV  Model — 6  Terms 
IS  0.015-n - — 

3 

%  0.010- 
{£. 

±  0.005- 

Q- 

2  0.000— ■ — - — ‘  — — 

C/1  . 

0) 

§  -0.005- 

LU 

¥-°-01°- 
CD 

2 -0.0 15 - 1 - f 


.00 


.02  .04  .06  .08  .10  .12 

Mean(ECI  ass  Prop/Hr)  Predicted 


6.47  <.0001 
10.93  <.0001 
8,45  <.0001 
4.15  0.0003 
6.23  <.0001 
-3.29  0.0028 


Interaction  Profiles 
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UAVRSW 
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0.1 

UAVPCIass 
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- 
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2 

MaxSteps 

3000  60009000 
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The  leverage  plots  for  the  two  UAV  scenario,  displayed  below,  show  the  relative 
effects  of  the  terms  in  the  preferred  model.  Time  has  a  positive  effect  here  as  opposed  to 
the  negative  effect  seen  in  the  single  UAV  model. 


Leverage  Plots  for  Two  UAV  6  Term  Model 
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The  model  produced  for  the  three  UAV  scenario,  displayed  below,  is  similar  to 
the  two  UAV  model  with  the  addition  of  a  sweep  width  squared  term  and  an  interaction 
between  time  and  probability  of  classification. 


Preferred  Model  for  Three  UAV — 7  Terms 

.  0.20- 


.05  .10  .15  .20 

Mean(ECIassProp/Hr)  Predicted  P<.0001 
RSq=0.90  RMSE=0.015 


Snmmaiy  of  Fit 

RSquare  0.90207 

RSquare  Adj  0.87465 

Root  Mean  Square  Error  0.01 4991 

Mean  of  Response  0.078766 

Observations  (or  Sum  Wgts)  33 

Analysis  of  Variance 

Source  DF  Sum  of  Squares  Mean  Square  F  Ratio 

Model  7  0.05175104  0.007393  32.8979 

Error  25  0.00561814  0.000225  Prob>F 

C.  Total  32  0.05736918  <.0001 

Paiametei  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

Intercept 

-0.051468 

0.013099 

-3.93 

0.0006 

UAVRSpeed 

0.0001505 

0.000028 

5.37 

<.0001 

UAVRSW 

0.0000117 

0.000001 

11.16 
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UAVPCIass 

0.0702681 

0.010497 

6.69 
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MaxSteps 
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0.00084 

-0.70 

0.4920 

(UA  VRSW-6058 .1 2)*(UAVPCIass-0.5) 

0.000014 
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2.75 

0.0110 

(UAVPCIass-0  5)’(MaxSteps-7 .00242) 

0.0083719 

0.004003 

2.09 

0.0468 

(UAVRSW-6058.1 2)*(UAVRSW-6058.1 2) 

-1  505e-9 

5.56e-10 

-2.71 

0.0121 

Residual  Plot  of  Preferred  Three  UAV  Model — 7  Terms 
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The  leverage  plots  for  the  three  UAV  scenario,  displayed  below,  show  the  relative 
effects  of  the  terms  in  the  preferred  model.  Sweep  width  is  most  significant  as  displayed 
by  the  steep  slope  of  the  sweep  width  plot.  Time  has  a  negative  effect  on  the  rate  of 
enemy  proportion  classified,  similar  to  the  effect  seen  in  the  one  UAV  model. 
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